Sep 12 02:35:59.165924 ip-10-0-130-57 systemd[1]: Starting Kubernetes Kubelet...
Sep 12 02:35:59.594382 ip-10-0-130-57 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 02:35:59.594382 ip-10-0-130-57 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 12 02:35:59.594382 ip-10-0-130-57 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 02:35:59.594382 ip-10-0-130-57 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 12 02:35:59.594382 ip-10-0-130-57 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 02:35:59.595270 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.594496    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 12 02:35:59.599790 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599768    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:35:59.599790 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599788    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:35:59.599790 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599795    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599799    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599804    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599808    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599813    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599817    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599820    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599825    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599829    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599833    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599837    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599841    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599845    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599850    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599854    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599858    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599861    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599865    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599869    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599873    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:35:59.600008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599877    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599880    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599884    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599888    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599892    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599895    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599899    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599903    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599907    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599912    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599916    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599919    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599925    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599931    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599937    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599941    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599946    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599950    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599954    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599958    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:35:59.600797 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599962    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599966    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599969    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599974    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599977    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599982    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599986    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599990    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.599994    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600001    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600005    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600008    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600012    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600016    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600020    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600024    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600028    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600033    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600037    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600041    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:35:59.601478 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600045    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600052    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600058    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600062    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600066    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600070    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600075    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600080    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600084    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600088    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600093    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600097    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600101    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600105    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600109    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600113    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600117    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600121    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600128    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:35:59.602001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600134    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600140    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600144    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600149    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600153    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600158    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600162    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600167    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600171    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600176    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600180    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600187    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600193    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600198    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600202    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600206    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600211    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:35:59.602454 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.600215    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:35:59.603582 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603565    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:35:59.603582 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603582    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603587    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603591    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603594    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603597    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603599    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603602    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603605    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603607    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603610    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603612    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603615    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603617    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603620    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603623    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603625    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603628    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603631    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603651    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:35:59.603652 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603656    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603658    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603661    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603664    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603667    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603669    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603672    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603674    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603677    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603679    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603682    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603688    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603693    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603696    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603699    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603702    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603705    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603708    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603710    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:35:59.604164 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603713    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603716    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603718    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603721    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603724    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603726    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603729    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603731    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603734    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603736    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603739    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603742    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603744    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603747    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603749    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603752    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603755    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603758    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603762    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603766    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:35:59.604623 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603768    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603771    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603774    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603776    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603778    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603782    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603785    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603788    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603791    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603795    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603802    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603805    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603808    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603811    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603814    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603816    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603819    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603821    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603824    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:35:59.605134 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603826    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603828    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603831    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603833    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603835    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603838    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603841    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603843    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603847    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603850    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603852    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603855    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603858    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603861    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603864    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603866    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603868    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603871    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603873    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:35:59.605596 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603876    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.603878    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604312    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604317    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604321    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604324    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604327    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604330    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604333    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604335    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604338    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604340    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604343    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604345    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604350    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604354    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604356    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604360    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604363    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604366    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:35:59.606078 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604369    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604372    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604375    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604378    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604381    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604383    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604386    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604389    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604392    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604394    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604397    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604399    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604402    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604404    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604406    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604409    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604412    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604414    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604416    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:35:59.606554 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604420    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604423    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604426    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604429    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604432    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604435    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604437    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604440    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604443    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604445    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604447    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604450    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604453    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604456    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604461    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604464    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604467    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604469    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604472    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604474    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:35:59.607039 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604477    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604479    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604481    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604484    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604486    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604489    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604491    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604494    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604496    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604498    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604501    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604503    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604506    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604508    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604510    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604513    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604515    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604518    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604520    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604523    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:35:59.607522 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604525    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604528    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604530    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604532    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604535    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604538    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604542    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604546    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604549    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604552    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604555    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604557    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604559    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604562    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604564    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604567    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604570    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604572    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604574    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:35:59.608021 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604577    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604580    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.604582    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604676    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604717    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604725    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604735    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604739    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604742    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604746    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604753    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604756    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604759    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604762    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604765    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604769    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604772    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604775    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604777    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604781    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604785    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604789    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604791    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 12 02:35:59.608480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604794    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604797    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604801    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604804    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604807    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604810    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604814    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604817    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604819    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604822    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604825    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604828    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604832    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604836    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604838    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604841    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604844    2579 flags.go:64] FLAG: --enable-server="true"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604847    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604851    2579 flags.go:64] FLAG: --event-burst="100"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604854    2579 flags.go:64] FLAG: --event-qps="50"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604857    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604860    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604863    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604867    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 12 02:35:59.609080 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604870    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604873    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604876    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604878    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604882    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604886    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604889    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604892    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604894    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604897    2579 flags.go:64] FLAG: --feature-gates=""
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604901    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604904    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604907    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604910    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604913    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604916    2579 flags.go:64] FLAG: --help="false"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604919    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604921    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604924    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604927    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604930    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604933    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604936    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 12 02:35:59.609681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604939    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604942    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604945    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604947    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604951    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604953    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604956    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604959    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604962    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604965    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604968    2579 flags.go:64] FLAG: --lock-file=""
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604970    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604973    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604977    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604982    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604986    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604989    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604992    2579 flags.go:64] FLAG: --logging-format="text"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604994    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.604997    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605000    2579 flags.go:64] FLAG: --manifest-url=""
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605003    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605007    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605010    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605014    2579 flags.go:64] FLAG: --max-pods="110"
Sep 12 02:35:59.610385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605017    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605020    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605022    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605025    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605028    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605031    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605034    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605041    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605044    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605047    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605050    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605056    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605061    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605064    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605067    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605070    2579 flags.go:64] FLAG: --port="10250"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605073    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605076    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-071be70ce58c16efd"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605079    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605082    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605085    2579 flags.go:64] FLAG: --register-node="true"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605088    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605092    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605095    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 12 02:35:59.611019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605098    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605100    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605104    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605107    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605110    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605113    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605115    2579 flags.go:64] FLAG: --runonce="false"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605118    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605121    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605124    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605126    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605129    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605132    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605135    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605138    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605141    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605144    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605147    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605149    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605152    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605156    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605159    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605164    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605167    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605170    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 12 02:35:59.611618 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605174    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605176    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605181    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605184    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605186    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605189    2579 flags.go:64] FLAG: --v="2"
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605193    2579 flags.go:64] FLAG: --version="false"
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605198    2579 flags.go:64] FLAG: --vmodule=""
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605201    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.605204    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605317    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605322    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605325    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605329    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605331    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605334    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605338    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605341    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605344    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605347    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605350    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:35:59.612238 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605353    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605355    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605358    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605360    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605363    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605365    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605369    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605371    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605374    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605378    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605381    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605384    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605387    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605391    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605394    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605397    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605399    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605402    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605405    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:35:59.612806 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605410    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605413    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605415    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605418    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605421    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605424    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605426    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605429    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605431    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605435    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605439    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605441    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605444    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605447    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605449    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605452    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605454    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605457    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605460    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605464    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:35:59.613313 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605466    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605469    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605471    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605474    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605476    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605479    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605482    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605485    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605487    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605490    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605493    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605495    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605498    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605501    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605503    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605506    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605508    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605511    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605513    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605516    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:35:59.613905 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605518    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605521    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605523    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605526    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605528    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605531    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605534    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605536    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605538    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605541    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605543    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605547    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605550    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605552    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605555    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605557    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605560    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605562    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605566    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:35:59.614399 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605568    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605571    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605574    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605576    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605579    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605581    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605583    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605586    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605589    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.605591    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.606511    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.614337    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 12 02:35:59.614871 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.614352    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 12 02:35:59.615309 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.615298    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 12 02:35:59.615340 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.615313    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 12 02:35:59.615387 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615379    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:35:59.615387 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615387    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615390    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615393    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615397    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615400    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615403    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615405    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615408    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615411    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615414    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615416    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615419    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615422    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615425    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615428    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615430    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615433    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615435    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615438    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615440    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:35:59.615440 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615443    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615445    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615448    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615451    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615453    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615456    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615458    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615461    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615464    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615467    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615470    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615472    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615475    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615477    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615479    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615482    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615484    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615487    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615489    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:35:59.615951 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615492    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615494    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615497    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615499    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615501    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615504    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615506    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615509    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615511    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615513    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615516    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615518    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615520    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615524    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615528    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615531    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615534    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615536    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615538    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615541    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:35:59.616446 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615544    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615546    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615549    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615552    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615555    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615557    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615560    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615562    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615565    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615568    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615572    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615575    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615578    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615582    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615585    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615588    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615591    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615594    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615596    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:35:59.616945 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615599    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615601    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615604    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615607    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615609    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615612    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615614    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615618    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615620    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615623    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615625    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615628    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615630    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615650    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615653    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615655    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615659    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615661    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615664    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:35:59.617395 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615666    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.615672    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615803    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615809    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615812    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615815    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615818    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615821    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615824    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615826    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615829    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615832    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615835    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615838    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:35:59.617859 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615841    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615843    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615846    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615849    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615851    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615854    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615857    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615859    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615862    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615865    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615867    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615870    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615872    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615875    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615877    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615880    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615882    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615885    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615887    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615890    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:35:59.618215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615892    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615895    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615897    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615900    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615902    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615905    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615907    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615909    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615912    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615914    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615917    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615919    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615921    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615924    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615926    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615929    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615931    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615933    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615936    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615938    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:35:59.618792 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615941    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615944    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615947    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615949    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615952    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615954    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615957    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615960    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615962    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615966    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615968    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615971    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615973    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615975    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615978    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615980    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615983    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615985    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615989    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:35:59.619293 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615993    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615996    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.615999    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616002    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616005    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616007    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616009    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616012    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616014    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616017    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616019    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616022    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616024    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616027    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616030    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616032    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616035    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616037    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616040    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616042    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:35:59.619765 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616044    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616047    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616049    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616052    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616055    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616057    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616060    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:35:59.616062    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.616067    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.616194    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.618543    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.619725    2579 server.go:1019] "Starting client certificate rotation"
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.619831    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 02:35:59.620261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.619862    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 02:35:59.622829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.622816    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 02:35:59.624224 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.624200    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 02:35:59.634419 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.634397    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 12 02:35:59.638413 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.638394    2579 log.go:25] "Validated CRI v1 image API"
Sep 12 02:35:59.639329 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.639313    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 12 02:35:59.642754 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.642732    2579 fs.go:135] Filesystem UUIDs: map[0c11fcbc-704a-4c95-a637-a57a67a0c251:/dev/nvme0n1p3 1ba4fd25-3486-4447-b229-7c1664c828f0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 12 02:35:59.642754 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.642749    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 12 02:35:59.648370 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.648354    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 02:35:59.648864 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.648751    2579 manager.go:216] Machine: {Timestamp:2026-09-12 02:35:59.646382906 +0000 UTC m=+0.370873653 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110579 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec293cfe3e08eee6a10539690b85fb99 SystemUUID:ec293cfe-3e08-eee6-a105-39690b85fb99 BootID:2749284b-ba9a-4adb-a8ef-c3b79bfc1331 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:7f:d6:83:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:7f:d6:83:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:b1:fc:9c:d2:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 12 02:35:59.648864 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.648861    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 12 02:35:59.648960 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.648939    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 12 02:35:59.652348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.652326    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 12 02:35:59.652479 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.652352    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 12 02:35:59.653197 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.653187    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 12 02:35:59.653237 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.653200    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 12 02:35:59.653237 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.653208    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 02:35:59.653237 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.653221    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 02:35:59.654470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.654461    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 12 02:35:59.659121 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659110    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 02:35:59.659230 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659220    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 12 02:35:59.659271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659250    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 12 02:35:59.659271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659269    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 12 02:35:59.659340 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659283    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 12 02:35:59.659340 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659294    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 12 02:35:59.659340 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659313    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 12 02:35:59.659675 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659662    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 02:35:59.659709 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.659689    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 02:35:59.661932 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.661917    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 12 02:35:59.664911 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.664896    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 12 02:35:59.664973 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.664927    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 12 02:35:59.666912 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666900    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666917    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666923    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666928    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666937    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666944    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666957    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666962    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666968    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 12 02:35:59.666971 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666973    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 12 02:35:59.667208 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666989    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 12 02:35:59.667208 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.666995    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 12 02:35:59.667902 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.667892    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 12 02:35:59.667902 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.667900    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 12 02:35:59.670682 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.670651    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 12 02:35:59.670786 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.670704    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 12 02:35:59.670786 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.670741    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 12 02:35:59.671844 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.671829    2579 server.go:1268] "Started kubelet"
Sep 12 02:35:59.671922 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.671905    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 12 02:35:59.671974 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.671921    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 12 02:35:59.672015 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.671999    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 12 02:35:59.672817 ip-10-0-130-57 systemd[1]: Started Kubernetes Kubelet.
Sep 12 02:35:59.673613 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.673587    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 12 02:35:59.674146 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.674131    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 12 02:35:59.676143 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.676126    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 12 02:35:59.680816 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.680798    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 12 02:35:59.681305 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.680143    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18d4723d5245b93b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:ip-10-0-130-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-09-12 02:35:59.671802171 +0000 UTC m=+0.396292919,LastTimestamp:2026-09-12 02:35:59.671802171 +0000 UTC m=+0.396292919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}"
Sep 12 02:35:59.681418 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.681405    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 12 02:35:59.681531 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.681509    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:35:59.682197 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682180    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 12 02:35:59.682197 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682184    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 12 02:35:59.682304 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682204    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 12 02:35:59.682304 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682289    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 12 02:35:59.682304 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682296    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 12 02:35:59.682415 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682379    2579 factory.go:153] Registering CRI-O factory
Sep 12 02:35:59.682415 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682392    2579 factory.go:223] Registration of the crio container factory successfully
Sep 12 02:35:59.682470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682435    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 12 02:35:59.682470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682442    2579 factory.go:55] Registering systemd factory
Sep 12 02:35:59.682470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682448    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 12 02:35:59.682470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682458    2579 factory.go:103] Registering Raw factory
Sep 12 02:35:59.682470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682469    2579 manager.go:1196] Started watching for new ooms in manager
Sep 12 02:35:59.682861 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.682849    2579 manager.go:349] Starting recovery of all containers
Sep 12 02:35:59.686232 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.686057    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 12 02:35:59.686232 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.686061    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 12 02:35:59.691564 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.691547    2579 manager.go:354] Recovery completed
Sep 12 02:35:59.695649 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.695617    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:35:59.697292 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.697273    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2gghp"
Sep 12 02:35:59.697539 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.697527    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:35:59.697578 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.697553    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:35:59.697578 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.697563    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:35:59.698103 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.698090    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 12 02:35:59.698103 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.698097    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:35:59.698103 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.698104    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 12 02:35:59.698250 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.698120    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 02:35:59.698878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.698863    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:35:59.698957 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.698887    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:35:59.698957 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.698897    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:35:59.699218 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.699122    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18d4723d53ce74e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:ip-10-0-130-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-09-12 02:35:59.697540326 +0000 UTC m=+0.422031073,LastTimestamp:2026-09-12 02:35:59.697540326 +0000 UTC m=+0.422031073,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}"
Sep 12 02:35:59.700299 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.700288    2579 policy_none.go:49] "None policy: Start"
Sep 12 02:35:59.700348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.700304    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 12 02:35:59.700348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.700313    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 12 02:35:59.702378 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.702365    2579 policy_none.go:47] "Start"
Sep 12 02:35:59.702454 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.702439    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2gghp"
Sep 12 02:35:59.708469 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.708409    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-57.ec2.internal.18d4723d53ceb96f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-57.ec2.internal,UID:ip-10-0-130-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-57.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-57.ec2.internal,},FirstTimestamp:2026-09-12 02:35:59.697557871 +0000 UTC m=+0.422048618,LastTimestamp:2026-09-12 02:35:59.697557871 +0000 UTC m=+0.422048618,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-57.ec2.internal,}"
Sep 12 02:35:59.729200 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.729188    2579 manager.go:337] "Starting Device Plugin manager"
Sep 12 02:35:59.729287 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.729221    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 12 02:35:59.729287 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.729231    2579 server.go:87] "Starting device plugin registration server"
Sep 12 02:35:59.729507 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.729495    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 12 02:35:59.729561 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.729511    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 12 02:35:59.729609 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.729592    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 12 02:35:59.729702 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.729686    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 12 02:35:59.729759 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.729703    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 12 02:35:59.730327 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.730293    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 12 02:35:59.730421 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.730348    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:35:59.784791 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.784764    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 12 02:35:59.786882 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.786016    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 12 02:35:59.786882 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.786055    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 12 02:35:59.786882 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.786074    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 12 02:35:59.786882 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.786107    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 12 02:35:59.788276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.788255    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:35:59.830129 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.830108    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:35:59.830944 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.830930    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:35:59.831002 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.830959    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:35:59.831002 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.830973    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:35:59.831002 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.830998    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.839263 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.839250    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.839305 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.839269    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-57.ec2.internal\": node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:35:59.862916 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.862857    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:35:59.886182 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.886150    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"]
Sep 12 02:35:59.886252 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.886240    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:35:59.887025 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.887008    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:35:59.887095 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.887039    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:35:59.887095 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.887050    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:35:59.889262 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889251    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:35:59.889369 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.889406 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889382    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:35:59.889928 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889913    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:35:59.890005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889930    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:35:59.890005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889944    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:35:59.890005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889955    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:35:59.890005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889968    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:35:59.890005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.889957    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:35:59.892105 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.892093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.892158 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.892117    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:35:59.892715 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.892700    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:35:59.892715 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.892720    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:35:59.892835 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.892734    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:35:59.917262 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.917238    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-57.ec2.internal\" not found" node="ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.921599 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.921582    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-57.ec2.internal\" not found" node="ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.963101 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:35:59.963067    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:35:59.984436 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.984414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03dcb3e205a55e01dbf2e10e467aa348-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"03dcb3e205a55e01dbf2e10e467aa348\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.984613 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.984449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03dcb3e205a55e01dbf2e10e467aa348-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"03dcb3e205a55e01dbf2e10e467aa348\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:35:59.984613 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:35:59.984475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/22695f442950d95e8040db7f0dc4719a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"22695f442950d95e8040db7f0dc4719a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.063563 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.063530    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.084915 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.084885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/22695f442950d95e8040db7f0dc4719a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"22695f442950d95e8040db7f0dc4719a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.085014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.084926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03dcb3e205a55e01dbf2e10e467aa348-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"03dcb3e205a55e01dbf2e10e467aa348\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.085014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.084952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03dcb3e205a55e01dbf2e10e467aa348-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"03dcb3e205a55e01dbf2e10e467aa348\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.085014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.084993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/03dcb3e205a55e01dbf2e10e467aa348-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"03dcb3e205a55e01dbf2e10e467aa348\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.085014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.084998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/22695f442950d95e8040db7f0dc4719a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-57.ec2.internal\" (UID: \"22695f442950d95e8040db7f0dc4719a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.085014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.085003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03dcb3e205a55e01dbf2e10e467aa348-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal\" (UID: \"03dcb3e205a55e01dbf2e10e467aa348\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.164417 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.164337    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.219733 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.219705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.224875 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.224630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:00.264460 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.264428    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.364874 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.364844    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.465453 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.465366    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.566003 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.565972    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.619543 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.619513    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 12 02:36:00.620140 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.619804    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 02:36:00.667075 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.667047    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.708271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.708226    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 02:31:01 +0000 UTC" deadline="2028-05-28 19:18:20.949512356 +0000 UTC"
Sep 12 02:36:00.708271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.708263    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14992h42m20.241252412s"
Sep 12 02:36:00.767575 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.767541    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.794753 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:36:00.794715    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dcb3e205a55e01dbf2e10e467aa348.slice/crio-9b442b805d4cc4e2dade54cf7a8776f4bdb7386eca144cbe457215101da54980 WatchSource:0}: Error finding container 9b442b805d4cc4e2dade54cf7a8776f4bdb7386eca144cbe457215101da54980: Status 404 returned error can't find the container with id 9b442b805d4cc4e2dade54cf7a8776f4bdb7386eca144cbe457215101da54980
Sep 12 02:36:00.795008 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:36:00.794984    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22695f442950d95e8040db7f0dc4719a.slice/crio-0e041430699d45ad9b8ecdeaadacd4002d8f2f4299320a3464ba47ff126a26a0 WatchSource:0}: Error finding container 0e041430699d45ad9b8ecdeaadacd4002d8f2f4299320a3464ba47ff126a26a0: Status 404 returned error can't find the container with id 0e041430699d45ad9b8ecdeaadacd4002d8f2f4299320a3464ba47ff126a26a0
Sep 12 02:36:00.798444 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.798425    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:36:00.808095 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.808074    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:00.808878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.808861    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:00.808951 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.808890    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:00.808951 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.808906    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:00.868322 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.868293    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:00.936279 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:00.936251    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:00.969132 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:00.969104    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-57.ec2.internal\" not found"
Sep 12 02:36:01.002187 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:01.002165    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:01.065103 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:01.065022    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:01.081995 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:01.081972    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:01.091374 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:01.091356    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 02:36:01.092186 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:01.092174    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"
Sep 12 02:36:01.102243 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:01.102226    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 02:36:03.116913 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.116878    2579 apiserver.go:52] "Watching apiserver"
Sep 12 02:36:03.124510 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.124482    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 12 02:36:03.124942 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.124909    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-plcc8","openshift-network-diagnostics/network-check-target-lslfl","openshift-network-operator/iptables-alerter-8v2l5","kube-system/global-pull-secret-syncer-kw4fm","kube-system/konnectivity-agent-fmp4l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq","openshift-image-registry/node-ca-85lmg","openshift-multus/multus-additional-cni-plugins-7sp86","openshift-multus/network-metrics-daemon-gncjn","openshift-ovn-kubernetes/ovnkube-node-grws5","kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fv7wq","openshift-dns/node-resolver-7flt6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal"]
Sep 12 02:36:03.127459 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.127431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.129694 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.129672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.130689 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.130662    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 12 02:36:03.130888 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.130872    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zv8jm\""
Sep 12 02:36:03.131052 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.131037    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 12 02:36:03.131151 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.131075    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 12 02:36:03.132051 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.132030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.132767 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.132749    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p2rbf\""
Sep 12 02:36:03.132890 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.132818    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 12 02:36:03.133056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.133042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 12 02:36:03.134905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.134696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:36:03.134905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.134757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.134905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.134758    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hthdg\""
Sep 12 02:36:03.134905 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.134823    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:03.135419 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.135397    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 12 02:36:03.135767 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.135749    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 12 02:36:03.137712 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.137695    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 12 02:36:03.140023 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.140007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.142543 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.142523    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dz99t\""
Sep 12 02:36:03.142635 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.142556    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 12 02:36:03.142635 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.142560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 12 02:36:03.142872 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.142855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.145186 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.145165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:03.145263 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.145234    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:03.145751 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.145729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbmph\""
Sep 12 02:36:03.145828 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.145800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 12 02:36:03.145988 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.145965    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 12 02:36:03.146152 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.145976    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 12 02:36:03.147408 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.147390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.149755 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.149731    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 12 02:36:03.150189 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150167    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 12 02:36:03.150268 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150191    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 02:36:03.150268 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150229    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s4stj\""
Sep 12 02:36:03.150268 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 12 02:36:03.150439 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150283    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 12 02:36:03.150439 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:03.150531 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.150478    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:03.150612 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-serviceca\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.150704 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-dbus\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.150704 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0f02bda-791d-4340-a758-2269e6c97bf4-hosts-file\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.150812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm749\" (UniqueName: \"kubernetes.io/projected/b0f02bda-791d-4340-a758-2269e6c97bf4-kube-api-access-pm749\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.150812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/464e05fa-e1e9-4da2-bf1c-13d879fd17e5-agent-certs\") pod \"konnectivity-agent-fmp4l\" (UID: \"464e05fa-e1e9-4da2-bf1c-13d879fd17e5\") " pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.150812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-kubelet-config\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.150812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/464e05fa-e1e9-4da2-bf1c-13d879fd17e5-konnectivity-ca\") pod \"konnectivity-agent-fmp4l\" (UID: \"464e05fa-e1e9-4da2-bf1c-13d879fd17e5\") " pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.151014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcgz\" (UniqueName: \"kubernetes.io/projected/5865a1be-eab7-4bc1-85c4-68ca0a243373-kube-api-access-wwcgz\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.151014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-host\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.151014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv4wf\" (UniqueName: \"kubernetes.io/projected/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-kube-api-access-bv4wf\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.151014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f02bda-791d-4340-a758-2269e6c97bf4-tmp-dir\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.151014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5865a1be-eab7-4bc1-85c4-68ca0a243373-iptables-alerter-script\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.151014 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.150965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5865a1be-eab7-4bc1-85c4-68ca0a243373-host-slash\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.151315 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.151028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.152918 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.152902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.155818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-29tld\""
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.155902    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.156004    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.156269    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.156293    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.156435    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.156455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 12 02:36:03.156766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.156521    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 12 02:36:03.158911 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.158891    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 12 02:36:03.159122 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.159080    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qtdnk\""
Sep 12 02:36:03.159203 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.159128    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 12 02:36:03.159819 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.159800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.162000 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.161961    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:36:03.162378 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.162360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 12 02:36:03.162806 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.162787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xvns8\""
Sep 12 02:36:03.168024 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.168004    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-98prc"
Sep 12 02:36:03.176878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.176844    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-98prc"
Sep 12 02:36:03.240144 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.240122    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 12 02:36:03.249739 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.249634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"03dcb3e205a55e01dbf2e10e467aa348","Type":"ContainerStarted","Data":"9b442b805d4cc4e2dade54cf7a8776f4bdb7386eca144cbe457215101da54980"}
Sep 12 02:36:03.251079 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.251054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"22695f442950d95e8040db7f0dc4719a","Type":"ContainerStarted","Data":"0e041430699d45ad9b8ecdeaadacd4002d8f2f4299320a3464ba47ff126a26a0"}
Sep 12 02:36:03.252274 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-cnibin\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.252274 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm749\" (UniqueName: \"kubernetes.io/projected/b0f02bda-791d-4340-a758-2269e6c97bf4-kube-api-access-pm749\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-kubelet\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-conf-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-host\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4pj\" (UniqueName: \"kubernetes.io/projected/91b0f569-893b-4909-af1e-823aaf54dfcc-kube-api-access-7j4pj\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxm5q\" (UniqueName: \"kubernetes.io/projected/4ba5bb9b-8684-4393-9747-2c1515ade783-kube-api-access-wxm5q\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.252438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cmv\" (UniqueName: \"kubernetes.io/projected/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-kube-api-access-n9cmv\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-run\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-registration-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/c0eba23f-369a-4cec-954d-d152d68cc4bd-kube-api-access-rwrz5\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.252516    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-cni-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.252599    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret podName:ff92f6c9-f9cb-4d8d-9056-c160e2de66cc nodeName:}" failed. No retries permitted until 2026-09-12 02:36:03.752563589 +0000 UTC m=+3.020597236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret") pod "global-pull-secret-syncer-kw4fm" (UID: "ff92f6c9-f9cb-4d8d-9056-c160e2de66cc") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-etc-kubernetes\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-cni-netd\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2n4\" (UniqueName: \"kubernetes.io/projected/11c7b4b6-0c5e-41af-9e15-d5736881babe-kube-api-access-jk2n4\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-socket-dir-parent\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-netns\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.252808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-cni-multus\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-kubernetes\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysctl-d\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-lib-modules\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5865a1be-eab7-4bc1-85c4-68ca0a243373-iptables-alerter-script\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-run-netns\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.252994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-env-overrides\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-systemd\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-dbus\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-system-cni-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-daemon-config\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91b0f569-893b-4909-af1e-823aaf54dfcc-tmp\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0f02bda-791d-4340-a758-2269e6c97bf4-hosts-file\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-dbus\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/464e05fa-e1e9-4da2-bf1c-13d879fd17e5-agent-certs\") pod \"konnectivity-agent-fmp4l\" (UID: \"464e05fa-e1e9-4da2-bf1c-13d879fd17e5\") " pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0f02bda-791d-4340-a758-2269e6c97bf4-hosts-file\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.253455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-k8s-cni-cncf-io\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-cni-bin\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-sys\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-sys-fs\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5865a1be-eab7-4bc1-85c4-68ca0a243373-iptables-alerter-script\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-systemd-units\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-etc-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-ovn\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovn-node-metrics-cert\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-modprobe-d\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv4wf\" (UniqueName: \"kubernetes.io/projected/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-kube-api-access-bv4wf\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-cnibin\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-cni-bin\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9mf\" (UniqueName: \"kubernetes.io/projected/a38160e4-6c6f-432d-8011-0f7472ed7d9c-kube-api-access-2s9mf\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-system-cni-dir\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysctl-conf\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253871    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-tuned\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-cni-binary-copy\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.253984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-kubelet\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-systemd\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-node-log\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-var-lib-kubelet\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-hostroot\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-multus-certs\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-log-socket\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-kubelet-config\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/464e05fa-e1e9-4da2-bf1c-13d879fd17e5-konnectivity-ca\") pod \"konnectivity-agent-fmp4l\" (UID: \"464e05fa-e1e9-4da2-bf1c-13d879fd17e5\") " pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-device-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.254958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-kubelet-config\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-slash\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-cni-binary-copy\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-var-lib-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-socket-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcgz\" (UniqueName: \"kubernetes.io/projected/5865a1be-eab7-4bc1-85c4-68ca0a243373-kube-api-access-wwcgz\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-host\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/464e05fa-e1e9-4da2-bf1c-13d879fd17e5-konnectivity-ca\") pod \"konnectivity-agent-fmp4l\" (UID: \"464e05fa-e1e9-4da2-bf1c-13d879fd17e5\") " pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f02bda-791d-4340-a758-2269e6c97bf4-tmp-dir\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5865a1be-eab7-4bc1-85c4-68ca0a243373-host-slash\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysconfig\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-serviceca\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-host\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovnkube-config\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.255654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovnkube-script-lib\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.256176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.254958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5865a1be-eab7-4bc1-85c4-68ca0a243373-host-slash\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.256176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.255122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-serviceca\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.256176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.255254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f02bda-791d-4340-a758-2269e6c97bf4-tmp-dir\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.258590 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.258458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/464e05fa-e1e9-4da2-bf1c-13d879fd17e5-agent-certs\") pod \"konnectivity-agent-fmp4l\" (UID: \"464e05fa-e1e9-4da2-bf1c-13d879fd17e5\") " pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.261465 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.261438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm749\" (UniqueName: \"kubernetes.io/projected/b0f02bda-791d-4340-a758-2269e6c97bf4-kube-api-access-pm749\") pod \"node-resolver-7flt6\" (UID: \"b0f02bda-791d-4340-a758-2269e6c97bf4\") " pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.262460 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.262434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv4wf\" (UniqueName: \"kubernetes.io/projected/3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de-kube-api-access-bv4wf\") pod \"node-ca-85lmg\" (UID: \"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de\") " pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.262664 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.262647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcgz\" (UniqueName: \"kubernetes.io/projected/5865a1be-eab7-4bc1-85c4-68ca0a243373-kube-api-access-wwcgz\") pod \"iptables-alerter-8v2l5\" (UID: \"5865a1be-eab7-4bc1-85c4-68ca0a243373\") " pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.355851 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-registration-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.355851 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/c0eba23f-369a-4cec-954d-d152d68cc4bd-kube-api-access-rwrz5\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-cni-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-etc-kubernetes\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-cni-netd\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-registration-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.355960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2n4\" (UniqueName: \"kubernetes.io/projected/11c7b4b6-0c5e-41af-9e15-d5736881babe-kube-api-access-jk2n4\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-socket-dir-parent\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-netns\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-cni-multus\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-cni-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-etc-kubernetes\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-kubernetes\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-cni-netd\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysctl-d\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-kubernetes\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-lib-modules\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-cni-multus\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-socket-dir-parent\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-run-netns\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-env-overrides\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysctl-d\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-systemd\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-netns\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-run-netns\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.356487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-lib-modules\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-systemd\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-system-cni-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-daemon-config\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91b0f569-893b-4909-af1e-823aaf54dfcc-tmp\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-system-cni-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-k8s-cni-cncf-io\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-cni-bin\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-k8s-cni-cncf-io\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-sys\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-sys-fs\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-systemd-units\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-sys\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.357325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-etc-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-cni-bin\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-ovn\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovn-node-metrics-cert\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-systemd-units\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-modprobe-d\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-sys-fs\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-cnibin\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-cni-bin\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-ovn\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-env-overrides\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-cnibin\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-etc-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9mf\" (UniqueName: \"kubernetes.io/projected/a38160e4-6c6f-432d-8011-0f7472ed7d9c-kube-api-access-2s9mf\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-daemon-config\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-cni-bin\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-modprobe-d\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-system-cni-dir\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.356995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysctl-conf\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-system-cni-dir\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-tuned\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-cni-binary-copy\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-kubelet\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysctl-conf\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-systemd\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.357209    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-run-systemd\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-node-log\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-var-lib-kubelet\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-node-log\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-var-lib-kubelet\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.358905 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.357322    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:36:03.857257252 +0000 UTC m=+3.125290878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-kubelet\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-hostroot\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-multus-certs\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-log-socket\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-hostroot\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-device-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-run-multus-certs\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-device-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-log-socket\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-cni-binary-copy\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-slash\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-slash\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-cni-binary-copy\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.359598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-var-lib-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-socket-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysconfig\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-var-lib-openvswitch\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-run-ovn-kubernetes\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-sysconfig\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.357997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4ba5bb9b-8684-4393-9747-2c1515ade783-socket-dir\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovnkube-config\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovnkube-script-lib\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-cnibin\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-kubelet\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-conf-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-host\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4pj\" (UniqueName: \"kubernetes.io/projected/91b0f569-893b-4909-af1e-823aaf54dfcc-kube-api-access-7j4pj\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c7b4b6-0c5e-41af-9e15-d5736881babe-cni-binary-copy\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxm5q\" (UniqueName: \"kubernetes.io/projected/4ba5bb9b-8684-4393-9747-2c1515ade783-kube-api-access-wxm5q\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-multus-conf-dir\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cmv\" (UniqueName: \"kubernetes.io/projected/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-kube-api-access-n9cmv\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c7b4b6-0c5e-41af-9e15-d5736881babe-cnibin\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-run\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-host-var-lib-kubelet\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovnkube-config\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-run\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a38160e4-6c6f-432d-8011-0f7472ed7d9c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91b0f569-893b-4909-af1e-823aaf54dfcc-host\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.358674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovnkube-script-lib\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.359744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91b0f569-893b-4909-af1e-823aaf54dfcc-tmp\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.359758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a38160e4-6c6f-432d-8011-0f7472ed7d9c-ovn-node-metrics-cert\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.360814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.359801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91b0f569-893b-4909-af1e-823aaf54dfcc-etc-tuned\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.364068 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.364047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2n4\" (UniqueName: \"kubernetes.io/projected/11c7b4b6-0c5e-41af-9e15-d5736881babe-kube-api-access-jk2n4\") pod \"multus-additional-cni-plugins-7sp86\" (UID: \"11c7b4b6-0c5e-41af-9e15-d5736881babe\") " pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.364349 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.364316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9mf\" (UniqueName: \"kubernetes.io/projected/a38160e4-6c6f-432d-8011-0f7472ed7d9c-kube-api-access-2s9mf\") pod \"ovnkube-node-grws5\" (UID: \"a38160e4-6c6f-432d-8011-0f7472ed7d9c\") " pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.364531 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.364511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrz5\" (UniqueName: \"kubernetes.io/projected/c0eba23f-369a-4cec-954d-d152d68cc4bd-kube-api-access-rwrz5\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:03.364952 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.364920    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:03.364952 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.364948    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:03.365103 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.364961    2579 projected.go:196] Error preparing data for projected volume kube-api-access-gzlc6 for pod openshift-network-diagnostics/network-check-target-lslfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:03.365103 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.365040    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6 podName:ad36f8e0-62b5-440b-adfd-a102899cac72 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:03.865023324 +0000 UTC m=+3.133056966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gzlc6" (UniqueName: "kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6") pod "network-check-target-lslfl" (UID: "ad36f8e0-62b5-440b-adfd-a102899cac72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:03.368829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.368769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cmv\" (UniqueName: \"kubernetes.io/projected/1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b-kube-api-access-n9cmv\") pod \"multus-plcc8\" (UID: \"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b\") " pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.369064 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.369050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4pj\" (UniqueName: \"kubernetes.io/projected/91b0f569-893b-4909-af1e-823aaf54dfcc-kube-api-access-7j4pj\") pod \"tuned-fv7wq\" (UID: \"91b0f569-893b-4909-af1e-823aaf54dfcc\") " pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.369214 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.369117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxm5q\" (UniqueName: \"kubernetes.io/projected/4ba5bb9b-8684-4393-9747-2c1515ade783-kube-api-access-wxm5q\") pod \"aws-ebs-csi-driver-node-vbczq\" (UID: \"4ba5bb9b-8684-4393-9747-2c1515ade783\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.408738 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.408712    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:03.440832 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.440791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-85lmg"
Sep 12 02:36:03.450124 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.450085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7flt6"
Sep 12 02:36:03.460948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.460922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8v2l5"
Sep 12 02:36:03.468711 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.468686    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:03.476514 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.476486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq"
Sep 12 02:36:03.483386 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.483360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-plcc8"
Sep 12 02:36:03.491537 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.491460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:03.499052 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.499029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7sp86"
Sep 12 02:36:03.503933 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.503914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fv7wq"
Sep 12 02:36:03.760528 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.760495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:03.760685 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.760657    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:03.760736 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.760723    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret podName:ff92f6c9-f9cb-4d8d-9056-c160e2de66cc nodeName:}" failed. No retries permitted until 2026-09-12 02:36:04.760708741 +0000 UTC m=+4.028742369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret") pod "global-pull-secret-syncer-kw4fm" (UID: "ff92f6c9-f9cb-4d8d-9056-c160e2de66cc") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:03.861723 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.861687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:03.861904 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.861842    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:03.861962 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.861916    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:36:04.861899203 +0000 UTC m=+4.129932845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:03.962954 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:03.962913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:03.963132 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.963070    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:03.963132 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.963122    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:03.963132 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.963133    2579 projected.go:196] Error preparing data for projected volume kube-api-access-gzlc6 for pod openshift-network-diagnostics/network-check-target-lslfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:03.963253 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:03.963192    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6 podName:ad36f8e0-62b5-440b-adfd-a102899cac72 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:04.963178949 +0000 UTC m=+4.231212573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzlc6" (UniqueName: "kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6") pod "network-check-target-lslfl" (UID: "ad36f8e0-62b5-440b-adfd-a102899cac72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:04.180968 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:04.180880    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 02:31:03 +0000 UTC" deadline="2028-03-16 15:47:12.459880738 +0000 UTC"
Sep 12 02:36:04.180968 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:04.180922    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13237h11m8.278962026s"
Sep 12 02:36:04.277870 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:36:04.275497    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80d30f_bdd5_486d_a9e0_fe60b4a4e0de.slice/crio-4bbbfb7c97dfe9fa7ed99ae4947f26055e1cc7c7496860ed5f4b3f72ba464340 WatchSource:0}: Error finding container 4bbbfb7c97dfe9fa7ed99ae4947f26055e1cc7c7496860ed5f4b3f72ba464340: Status 404 returned error can't find the container with id 4bbbfb7c97dfe9fa7ed99ae4947f26055e1cc7c7496860ed5f4b3f72ba464340
Sep 12 02:36:04.278542 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:36:04.278468    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f02bda_791d_4340_a758_2269e6c97bf4.slice/crio-d0d1c67a1de1498abe104460731256db297f90811850e83d7f7608d241f102f9 WatchSource:0}: Error finding container d0d1c67a1de1498abe104460731256db297f90811850e83d7f7608d241f102f9: Status 404 returned error can't find the container with id d0d1c67a1de1498abe104460731256db297f90811850e83d7f7608d241f102f9
Sep 12 02:36:04.281411 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:36:04.281388    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b0f569_893b_4909_af1e_823aaf54dfcc.slice/crio-a48b04310bbc361fed3a32da545bd8876adff5d44ae024fcc5a19840df79f2ac WatchSource:0}: Error finding container a48b04310bbc361fed3a32da545bd8876adff5d44ae024fcc5a19840df79f2ac: Status 404 returned error can't find the container with id a48b04310bbc361fed3a32da545bd8876adff5d44ae024fcc5a19840df79f2ac
Sep 12 02:36:04.285268 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:36:04.285181    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba5bb9b_8684_4393_9747_2c1515ade783.slice/crio-6ad2344aa7afc3e32a44805729f969a0a38204c82d42cd355ae3236a62be0481 WatchSource:0}: Error finding container 6ad2344aa7afc3e32a44805729f969a0a38204c82d42cd355ae3236a62be0481: Status 404 returned error can't find the container with id 6ad2344aa7afc3e32a44805729f969a0a38204c82d42cd355ae3236a62be0481
Sep 12 02:36:04.767905 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:04.767826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:04.768048 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.767996    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:04.768125 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.768049    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret podName:ff92f6c9-f9cb-4d8d-9056-c160e2de66cc nodeName:}" failed. No retries permitted until 2026-09-12 02:36:06.768033887 +0000 UTC m=+6.036067511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret") pod "global-pull-secret-syncer-kw4fm" (UID: "ff92f6c9-f9cb-4d8d-9056-c160e2de66cc") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:04.868487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:04.868446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:04.868665 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.868606    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:04.868734 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.868671    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:36:06.868650879 +0000 UTC m=+6.136684527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:04.969119 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:04.968822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:04.969119 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.969040    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:04.969119 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.969060    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:04.969119 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.969072    2579 projected.go:196] Error preparing data for projected volume kube-api-access-gzlc6 for pod openshift-network-diagnostics/network-check-target-lslfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:04.969452 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:04.969145    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6 podName:ad36f8e0-62b5-440b-adfd-a102899cac72 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:06.969128502 +0000 UTC m=+6.237162141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzlc6" (UniqueName: "kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6") pod "network-check-target-lslfl" (UID: "ad36f8e0-62b5-440b-adfd-a102899cac72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:05.181810 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.181719    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 02:31:03 +0000 UTC" deadline="2028-04-04 07:46:30.171587599 +0000 UTC"
Sep 12 02:36:05.181810 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.181779    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13685h10m24.989812765s"
Sep 12 02:36:05.245566 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.245532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:05.245739 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:05.245665    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:05.246107 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.246069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:05.246220 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:05.246187    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:05.246285 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.246252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:05.246349 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:05.246331    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:05.268204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.268168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-plcc8" event={"ID":"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b","Type":"ContainerStarted","Data":"3f82908841fcf66ff3d8fbd868c8c641bf4219363f88d02e765845184dd5af72"}
Sep 12 02:36:05.275356 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.275296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerStarted","Data":"0f34c09382c2ac9a443b678a8264ac619eef55660c3622dbf51e06b046ac1c14"}
Sep 12 02:36:05.281312 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.281252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8v2l5" event={"ID":"5865a1be-eab7-4bc1-85c4-68ca0a243373","Type":"ContainerStarted","Data":"a95c8b4bd3c9587b195c1bd43685c0e71f54f5d3b0679a2e7918f173e38c6ac5"}
Sep 12 02:36:05.294958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.294891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq" event={"ID":"4ba5bb9b-8684-4393-9747-2c1515ade783","Type":"ContainerStarted","Data":"6ad2344aa7afc3e32a44805729f969a0a38204c82d42cd355ae3236a62be0481"}
Sep 12 02:36:05.297901 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.297832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7flt6" event={"ID":"b0f02bda-791d-4340-a758-2269e6c97bf4","Type":"ContainerStarted","Data":"d0d1c67a1de1498abe104460731256db297f90811850e83d7f7608d241f102f9"}
Sep 12 02:36:05.305926 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.305897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-85lmg" event={"ID":"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de","Type":"ContainerStarted","Data":"4bbbfb7c97dfe9fa7ed99ae4947f26055e1cc7c7496860ed5f4b3f72ba464340"}
Sep 12 02:36:05.331691 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.331654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"19a4f1398fd5d6436738eb744aa4328e0c2a7dda04ccdd67dda33bb30afec8f5"}
Sep 12 02:36:05.344142 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.343690    2579 generic.go:356] "Generic (PLEG): container finished" podID="03dcb3e205a55e01dbf2e10e467aa348" containerID="59a5d54b6f5129fd432db0a11c04fd59d7a7c19e0b8502ec6a2396d0eeb79a49" exitCode=0
Sep 12 02:36:05.344142 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.343779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"03dcb3e205a55e01dbf2e10e467aa348","Type":"ContainerDied","Data":"59a5d54b6f5129fd432db0a11c04fd59d7a7c19e0b8502ec6a2396d0eeb79a49"}
Sep 12 02:36:05.360544 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.360218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" event={"ID":"22695f442950d95e8040db7f0dc4719a","Type":"ContainerStarted","Data":"1c974dfe90fb9129ee695282276fca04565b15f8c4edfac2205ca11fbaec822d"}
Sep 12 02:36:05.378263 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.378233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmp4l" event={"ID":"464e05fa-e1e9-4da2-bf1c-13d879fd17e5","Type":"ContainerStarted","Data":"8f6ffc0e2892dbf4a6a5f1bb652c552f849333280f609b25cd1c310cc4f5b42e"}
Sep 12 02:36:05.391292 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:05.391259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fv7wq" event={"ID":"91b0f569-893b-4909-af1e-823aaf54dfcc","Type":"ContainerStarted","Data":"a48b04310bbc361fed3a32da545bd8876adff5d44ae024fcc5a19840df79f2ac"}
Sep 12 02:36:06.410249 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:06.410158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" event={"ID":"03dcb3e205a55e01dbf2e10e467aa348","Type":"ContainerStarted","Data":"a7636455951b70b5ef54b809976cef880ef867a120f443ad712b5ec7d47c1172"}
Sep 12 02:36:06.426128 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:06.425361    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-57.ec2.internal" podStartSLOduration=4.425340138 podStartE2EDuration="4.425340138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:05.37308449 +0000 UTC m=+4.641118130" watchObservedRunningTime="2026-09-12 02:36:06.425340138 +0000 UTC m=+5.693373784"
Sep 12 02:36:06.794206 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:06.794168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:06.794419 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.794394    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:06.794571 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.794464    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret podName:ff92f6c9-f9cb-4d8d-9056-c160e2de66cc nodeName:}" failed. No retries permitted until 2026-09-12 02:36:10.794446004 +0000 UTC m=+10.062479649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret") pod "global-pull-secret-syncer-kw4fm" (UID: "ff92f6c9-f9cb-4d8d-9056-c160e2de66cc") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:06.894992 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:06.894954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:06.895188 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.895170    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:06.895253 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.895236    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:36:10.89521795 +0000 UTC m=+10.163251579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:06.995956 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:06.995923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:06.996158 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.996078    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:06.996158 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.996118    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:06.996158 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.996129    2579 projected.go:196] Error preparing data for projected volume kube-api-access-gzlc6 for pod openshift-network-diagnostics/network-check-target-lslfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:06.996326 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:06.996196    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6 podName:ad36f8e0-62b5-440b-adfd-a102899cac72 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:10.996176252 +0000 UTC m=+10.264209891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzlc6" (UniqueName: "kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6") pod "network-check-target-lslfl" (UID: "ad36f8e0-62b5-440b-adfd-a102899cac72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:07.246015 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:07.245978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:07.246662 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:07.246623    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:07.250249 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:07.250226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:07.250405 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:07.250381    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:07.250491 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:07.250468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:07.250612 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:07.250592    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:09.243209 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:09.242903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:09.243713 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:09.243289    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:09.243713 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:09.243657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:09.243809 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:09.243748    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:09.243855 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:09.243821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:09.243936 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:09.243913    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:10.831856 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:10.831772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:10.832338 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:10.831944    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:10.832338 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:10.832028    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret podName:ff92f6c9-f9cb-4d8d-9056-c160e2de66cc nodeName:}" failed. No retries permitted until 2026-09-12 02:36:18.832006968 +0000 UTC m=+18.100040617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret") pod "global-pull-secret-syncer-kw4fm" (UID: "ff92f6c9-f9cb-4d8d-9056-c160e2de66cc") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:10.932669 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:10.932592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:10.932836 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:10.932761    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:10.932908 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:10.932840    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:36:18.932819838 +0000 UTC m=+18.200853468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:11.033856 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:11.033270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:11.033856 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:11.033465    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:11.033856 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:11.033485    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:11.033856 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:11.033494    2579 projected.go:196] Error preparing data for projected volume kube-api-access-gzlc6 for pod openshift-network-diagnostics/network-check-target-lslfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:11.033856 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:11.033550    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6 podName:ad36f8e0-62b5-440b-adfd-a102899cac72 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:19.033536428 +0000 UTC m=+18.301570052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzlc6" (UniqueName: "kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6") pod "network-check-target-lslfl" (UID: "ad36f8e0-62b5-440b-adfd-a102899cac72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:11.244480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:11.244442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:11.244662 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:11.244565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:11.244662 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:11.244570    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:11.244662 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:11.244589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:11.244784 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:11.244702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:11.244820 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:11.244790    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:13.243538 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:13.243014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:13.243538 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:13.243022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:13.243538 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:13.243169    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:13.243538 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:13.243022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:13.243538 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:13.243227    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:13.243538 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:13.243280    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:15.243549 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:15.243516    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:15.243549 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:15.243547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:15.243896 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:15.243537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:15.243896 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:15.243644    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:15.243896 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:15.243705    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:15.243896 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:15.243833    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:17.243548 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:17.243509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:17.244027 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:17.243509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:17.244027 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:17.243658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:17.244027 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:17.243509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:17.244027 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:17.243736    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:17.244027 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:17.243832    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:18.893271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:18.893232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:18.893845 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:18.893418    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:18.893845 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:18.893501    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret podName:ff92f6c9-f9cb-4d8d-9056-c160e2de66cc nodeName:}" failed. No retries permitted until 2026-09-12 02:36:34.893481269 +0000 UTC m=+34.161514914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret") pod "global-pull-secret-syncer-kw4fm" (UID: "ff92f6c9-f9cb-4d8d-9056-c160e2de66cc") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:18.993606 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:18.993581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:18.993723 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:18.993691    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:18.993767 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:18.993737    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:36:34.993723742 +0000 UTC m=+34.261757367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:19.094888 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:19.094853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:19.095069 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:19.095001    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:19.095069 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:19.095025    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:19.095069 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:19.095035    2579 projected.go:196] Error preparing data for projected volume kube-api-access-gzlc6 for pod openshift-network-diagnostics/network-check-target-lslfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:19.095251 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:19.095116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6 podName:ad36f8e0-62b5-440b-adfd-a102899cac72 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:35.0950812 +0000 UTC m=+34.363114826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzlc6" (UniqueName: "kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6") pod "network-check-target-lslfl" (UID: "ad36f8e0-62b5-440b-adfd-a102899cac72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:19.243031 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:19.242999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:19.243220 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:19.243046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:19.243220 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:19.243129    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:19.243220 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:19.243165    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:19.243220 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:19.243198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:19.243374 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:19.243256    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:21.244910 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:21.244414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:21.244910 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:21.244530    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:21.244910 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:21.244536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:21.244910 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:21.244652    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:21.244910 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:21.244705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:21.244910 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:21.244763    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:23.247407 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.247131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:23.248359 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.247178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:23.248359 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:23.247462    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:23.248359 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:23.247541    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:23.248359 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.247217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:23.248359 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:23.247622    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:23.450835 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.450803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"dbe15c994521686801470a1f1fba8a4bdf4226ef1a06250c9c7d9f8f2ea27dfc"}
Sep 12 02:36:23.451178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.450879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"8c5a7855648eb4186fe93f0bce006a07cbb402a87b448fb063d53953bbbbecdd"}
Sep 12 02:36:23.451178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.450909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"6963a8f11bbb196afdf1d8b33e22db6361ae1fca9cf9cf69fe7966e42ade70e7"}
Sep 12 02:36:23.451178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.450940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"3a1fd9f7593daecf7fe0f185ec9ebcf5b7c0f0d359f04de8d4fc8361c128d59e"}
Sep 12 02:36:23.451178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.450967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"9f4fc11765417b30bd551408e9e63677f93b36b9863b65bb9dc8cc115fb7e035"}
Sep 12 02:36:23.451178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.450992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"f4988608d0dd1b44ffff8e98390d7735ee3ed7536bb5a0b5189e180b7975445b"}
Sep 12 02:36:23.452271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.452246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmp4l" event={"ID":"464e05fa-e1e9-4da2-bf1c-13d879fd17e5","Type":"ContainerStarted","Data":"a74a8e93592c599bc37b939b13915b1d2c6ec42cb1719afd2bd1ef84d9b56779"}
Sep 12 02:36:23.453742 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.453686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fv7wq" event={"ID":"91b0f569-893b-4909-af1e-823aaf54dfcc","Type":"ContainerStarted","Data":"58e7e95515082b8b92b4fbdc7a4fae5ea5d4d03f4f1c22560b8ed937ff031b52"}
Sep 12 02:36:23.455064 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.455041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-plcc8" event={"ID":"1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b","Type":"ContainerStarted","Data":"72ef2e92c9880d04eeb490bb4d85dc5f220e2d9413e8fc8c537e3e3f7da51eed"}
Sep 12 02:36:23.456398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.456379    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c7b4b6-0c5e-41af-9e15-d5736881babe" containerID="0ed72a3599b7989eba2059324731be7259bc9924aa2fee8bad56f4f997ef9e43" exitCode=0
Sep 12 02:36:23.456484 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.456441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerDied","Data":"0ed72a3599b7989eba2059324731be7259bc9924aa2fee8bad56f4f997ef9e43"}
Sep 12 02:36:23.457913 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.457865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq" event={"ID":"4ba5bb9b-8684-4393-9747-2c1515ade783","Type":"ContainerStarted","Data":"8a36c8de289e8bd84b76b75aa8463e5be4be665253609aafa42ba45860be526c"}
Sep 12 02:36:23.459541 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.459516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7flt6" event={"ID":"b0f02bda-791d-4340-a758-2269e6c97bf4","Type":"ContainerStarted","Data":"869f3d41eba8ae65ee445f10476f026ea437e18db97c9261a74cc6975b9d0b40"}
Sep 12 02:36:23.462775 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.462753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-85lmg" event={"ID":"3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de","Type":"ContainerStarted","Data":"9ed0d296284a944c9c21f396dd54fdded52b1cdb88a861aa6d0ac53c41236a3e"}
Sep 12 02:36:23.468480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.468438    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fmp4l" podStartSLOduration=7.772706249 podStartE2EDuration="22.468425847s" totalImagesPullingTime="14.695719598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:23.468358487 +0000 UTC m=+22.736392134" watchObservedRunningTime="2026-09-12 02:36:23.468425847 +0000 UTC m=+22.736459494"
Sep 12 02:36:23.468909 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.468888    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-57.ec2.internal" podStartSLOduration=21.468883607 podStartE2EDuration="21.468883607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:06.426058568 +0000 UTC m=+5.694092215" watchObservedRunningTime="2026-09-12 02:36:23.468883607 +0000 UTC m=+22.736917253"
Sep 12 02:36:23.504356 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.504265    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fv7wq" podStartSLOduration=4.30682218 podStartE2EDuration="22.504248093s" totalImagesPullingTime="18.197425913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:23.504112343 +0000 UTC m=+22.772145984" watchObservedRunningTime="2026-09-12 02:36:23.504248093 +0000 UTC m=+22.772281732"
Sep 12 02:36:23.537980 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.537924    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-85lmg" podStartSLOduration=4.34615539 podStartE2EDuration="22.537907498s" totalImagesPullingTime="18.191752108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:23.518750479 +0000 UTC m=+22.786784128" watchObservedRunningTime="2026-09-12 02:36:23.537907498 +0000 UTC m=+22.805941145"
Sep 12 02:36:23.564986 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.564896    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-plcc8" podStartSLOduration=4.334919957 podStartE2EDuration="22.564871476s" totalImagesPullingTime="18.229951519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:23.564860966 +0000 UTC m=+22.832894624" watchObservedRunningTime="2026-09-12 02:36:23.564871476 +0000 UTC m=+22.832905120"
Sep 12 02:36:23.567291 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.567243    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7flt6" podStartSLOduration=4.375633655 podStartE2EDuration="22.567226964s" totalImagesPullingTime="18.191593309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:23.537715644 +0000 UTC m=+22.805749290" watchObservedRunningTime="2026-09-12 02:36:23.567226964 +0000 UTC m=+22.835260612"
Sep 12 02:36:23.833862 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:23.833825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:24.063023 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:24.062853    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 12 02:36:24.195644 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:24.195538    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T02:36:24.063015196Z","UUID":"ac9fa9a4-a3e2-4861-af85-87d1e35f5e57","Handler":null,"Name":"","Endpoint":""}
Sep 12 02:36:24.198381 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:24.198353    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 12 02:36:24.198552 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:24.198394    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 12 02:36:24.467107 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:24.467069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq" event={"ID":"4ba5bb9b-8684-4393-9747-2c1515ade783","Type":"ContainerStarted","Data":"120658337bcb9ca7c1ead6c46222a596ff842f3939b7009649f0c3f76bc29454"}
Sep 12 02:36:24.468439 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:24.468408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8v2l5" event={"ID":"5865a1be-eab7-4bc1-85c4-68ca0a243373","Type":"ContainerStarted","Data":"b13bfd30f8da66008ed2402c5bcc0b4edbb33233734685e7be7f76b7431188e5"}
Sep 12 02:36:24.482745 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:24.482691    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8v2l5" podStartSLOduration=5.303339458 podStartE2EDuration="23.482675422s" totalImagesPullingTime="18.179335964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:24.482502899 +0000 UTC m=+23.750536546" watchObservedRunningTime="2026-09-12 02:36:24.482675422 +0000 UTC m=+23.750709069"
Sep 12 02:36:25.245083 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:25.243815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:25.245083 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:25.243959    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:25.245083 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:25.244396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:25.245083 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:25.244491    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:25.245083 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:25.244646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:25.245083 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:25.244734    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:25.473255 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:25.473216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq" event={"ID":"4ba5bb9b-8684-4393-9747-2c1515ade783","Type":"ContainerStarted","Data":"c2e848e403eaa936314015a843ebee485254be095a7aa2d7a541f1ea85bde1c2"}
Sep 12 02:36:25.477278 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:25.477246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"bbaf5456db1e64b16b6bbda83b3e3f92cf914e2c9d552cc6215dac4e057323b6"}
Sep 12 02:36:25.490985 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:25.490909    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbczq" podStartSLOduration=4.037773417 podStartE2EDuration="24.490883649s" totalImagesPullingTime="20.453110232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:25.489780941 +0000 UTC m=+24.757814598" watchObservedRunningTime="2026-09-12 02:36:25.490883649 +0000 UTC m=+24.758917298"
Sep 12 02:36:26.911416 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:26.911379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:26.912040 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:26.912021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:27.243807 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:27.243766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:27.244005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:27.243766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:27.244005 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:27.243915    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:27.244139 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:27.244001    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:27.244139 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:27.243775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:27.244139 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:27.244082    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:27.482477 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:27.482450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fmp4l"
Sep 12 02:36:28.486274 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:28.485968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerStarted","Data":"d0db58873ae6073683e329c8aff6f403f1537093a64387784a4ab4642ffcd858"}
Sep 12 02:36:28.490454 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:28.490411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" event={"ID":"a38160e4-6c6f-432d-8011-0f7472ed7d9c","Type":"ContainerStarted","Data":"ca7eb79f0c66a23287e1a43ff3162c3ef5c42a7da6e95c15f41aa9308a4586a5"}
Sep 12 02:36:28.536012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:28.535960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-grws5" podStartSLOduration=9.281379213 podStartE2EDuration="27.53594591s" totalImagesPullingTime="18.254566697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:28.534865751 +0000 UTC m=+27.802899398" watchObservedRunningTime="2026-09-12 02:36:28.53594591 +0000 UTC m=+27.803979557"
Sep 12 02:36:29.243452 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.243421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:29.243604 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.243421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:29.243667 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:29.243644    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:29.243699 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.243428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:29.243758 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:29.243522    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:29.243811 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:29.243758    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:29.495440 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.495349    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c7b4b6-0c5e-41af-9e15-d5736881babe" containerID="d0db58873ae6073683e329c8aff6f403f1537093a64387784a4ab4642ffcd858" exitCode=0
Sep 12 02:36:29.495440 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.495421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerDied","Data":"d0db58873ae6073683e329c8aff6f403f1537093a64387784a4ab4642ffcd858"}
Sep 12 02:36:29.496256 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.496237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:29.496335 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.496322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:29.496398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.496389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:29.517890 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.517734    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:29.526846 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:29.526751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:36:30.362915 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:30.362880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kw4fm"]
Sep 12 02:36:30.363088 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:30.363005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:30.363176 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:30.363139    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:30.366985 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:30.366946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lslfl"]
Sep 12 02:36:30.367179 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:30.367120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:30.367242 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:30.367223    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:30.371112 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:30.371067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gncjn"]
Sep 12 02:36:30.371262 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:30.371212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:30.371338 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:30.371319    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:30.500526 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:30.500482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerStarted","Data":"a1c29a4b320aa1d77f9f2024f287da672fd0735a788404b9bd98ec66af8894c9"}
Sep 12 02:36:31.506295 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:31.506256    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c7b4b6-0c5e-41af-9e15-d5736881babe" containerID="a1c29a4b320aa1d77f9f2024f287da672fd0735a788404b9bd98ec66af8894c9" exitCode=0
Sep 12 02:36:31.506750 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:31.506346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerDied","Data":"a1c29a4b320aa1d77f9f2024f287da672fd0735a788404b9bd98ec66af8894c9"}
Sep 12 02:36:32.244391 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:32.244170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:32.244548 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:32.244196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:32.244548 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:32.244505    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:32.244675 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:32.244575    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:32.244675 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:32.244196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:32.244775 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:32.244676    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:32.511454 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:32.511359    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c7b4b6-0c5e-41af-9e15-d5736881babe" containerID="aa3be9607065e9c12f97474dc1494aa011ec8eb09c2050efc41bd48b0721080a" exitCode=0
Sep 12 02:36:32.511454 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:32.511423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerDied","Data":"aa3be9607065e9c12f97474dc1494aa011ec8eb09c2050efc41bd48b0721080a"}
Sep 12 02:36:34.243659 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:34.243620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:34.244170 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:34.243623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:34.244170 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:34.243785    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:36:34.244170 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:34.243633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:34.244170 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:34.243852    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kw4fm" podUID="ff92f6c9-f9cb-4d8d-9056-c160e2de66cc"
Sep 12 02:36:34.244170 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:34.243889    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lslfl" podUID="ad36f8e0-62b5-440b-adfd-a102899cac72"
Sep 12 02:36:34.915511 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:34.915468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:34.915858 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:34.915626    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:34.915858 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:34.915681    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret podName:ff92f6c9-f9cb-4d8d-9056-c160e2de66cc nodeName:}" failed. No retries permitted until 2026-09-12 02:37:06.915667294 +0000 UTC m=+66.183700931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret") pod "global-pull-secret-syncer-kw4fm" (UID: "ff92f6c9-f9cb-4d8d-9056-c160e2de66cc") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:35.016663 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.016628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:35.016843 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.016809    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:35.016973 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.016896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:37:07.016875412 +0000 UTC m=+66.284909059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:35.117355 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.117310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:35.117536 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.117497    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:35.117536 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.117521    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:35.117536 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.117532    2579 projected.go:196] Error preparing data for projected volume kube-api-access-gzlc6 for pod openshift-network-diagnostics/network-check-target-lslfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:35.117661 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.117587    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6 podName:ad36f8e0-62b5-440b-adfd-a102899cac72 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:07.117573885 +0000 UTC m=+66.385607510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzlc6" (UniqueName: "kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6") pod "network-check-target-lslfl" (UID: "ad36f8e0-62b5-440b-adfd-a102899cac72") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:35.491635 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.491599    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-57.ec2.internal" event="NodeReady"
Sep 12 02:36:35.492045 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.491768    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 12 02:36:35.523922 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.523891    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85d54dd6dd-xkg4p"]
Sep 12 02:36:35.532313 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.532237    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"]
Sep 12 02:36:35.532459 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.532410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.535694 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.535667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jtp6k\""
Sep 12 02:36:35.535694 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.535689    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 12 02:36:35.535881 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.535732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 12 02:36:35.535881 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.535688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 12 02:36:35.540987 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.540965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:35.541054 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.541000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 12 02:36:35.542222 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.542197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"]
Sep 12 02:36:35.543257 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.542974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d54dd6dd-xkg4p"]
Sep 12 02:36:35.543710 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.543689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 12 02:36:35.543925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.543909    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 12 02:36:35.544176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.544155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-88r85\""
Sep 12 02:36:35.544649 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.544630    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6znn9"]
Sep 12 02:36:35.550407 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.550382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hrs62"]
Sep 12 02:36:35.550529 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.550512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.553163 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.553140    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 12 02:36:35.553163 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.553160    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 12 02:36:35.553310 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.553165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4279k\""
Sep 12 02:36:35.560732 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.560680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hrs62"]
Sep 12 02:36:35.560732 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.560726    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6znn9"]
Sep 12 02:36:35.560983 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.560817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:35.564176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.563431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-96kxp\""
Sep 12 02:36:35.564176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.563815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 12 02:36:35.564176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.563949    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 12 02:36:35.564176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.564035    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 12 02:36:35.621788 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.621689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-bound-sa-token\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.621788 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.621742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-installation-pull-secrets\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.621788 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.621778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb403fd-1ca8-4352-a7c7-ff62805c099c-tmp-dir\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.622065 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.621851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-certificates\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.622065 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.621871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-trusted-ca\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.622291 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.621915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:35.622368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.622338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb403fd-1ca8-4352-a7c7-ff62805c099c-config-volume\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.622420 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.622380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.622456 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.622440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eee61980-927f-493a-8e93-fafe1447d816-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:35.622519 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.622504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.622574 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.622565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvlsf\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-kube-api-access-kvlsf\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.622626 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.622595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzcft\" (UniqueName: \"kubernetes.io/projected/4cb403fd-1ca8-4352-a7c7-ff62805c099c-kube-api-access-vzcft\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.622727 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.622709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-image-registry-private-configuration\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.624285 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.624229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56373d50-df04-4236-80c5-9de7b5c6dc24-ca-trust-extracted\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.647669 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.647626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:35.724926 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.724876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd1fe03-285d-4060-9a10-10e9c46b5566-cert\") pod \"ingress-canary-hrs62\" (UID: \"7dd1fe03-285d-4060-9a10-10e9c46b5566\") " pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:35.725167 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.724961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eee61980-927f-493a-8e93-fafe1447d816-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:35.725167 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725167 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvlsf\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-kube-api-access-kvlsf\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzcft\" (UniqueName: \"kubernetes.io/projected/4cb403fd-1ca8-4352-a7c7-ff62805c099c-kube-api-access-vzcft\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.725213    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-image-registry-private-configuration\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56373d50-df04-4236-80c5-9de7b5c6dc24-ca-trust-extracted\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-bound-sa-token\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-installation-pull-secrets\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb403fd-1ca8-4352-a7c7-ff62805c099c-tmp-dir\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.725348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-certificates\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725725 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-trusted-ca\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.725725 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfkp\" (UniqueName: \"kubernetes.io/projected/7dd1fe03-285d-4060-9a10-10e9c46b5566-kube-api-access-6sfkp\") pod \"ingress-canary-hrs62\" (UID: \"7dd1fe03-285d-4060-9a10-10e9c46b5566\") " pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:35.725725 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:35.725725 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb403fd-1ca8-4352-a7c7-ff62805c099c-config-volume\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.725725 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.725928 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.725887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eee61980-927f-493a-8e93-fafe1447d816-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:35.725928 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.725238    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:36:35.726017 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.725975    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:36.22595922 +0000 UTC m=+35.493992844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:36:35.726272 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.725617    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:35.726272 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.726215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56373d50-df04-4236-80c5-9de7b5c6dc24-ca-trust-extracted\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.726272 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.726246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb403fd-1ca8-4352-a7c7-ff62805c099c-config-volume\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.726272 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.725699    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:36:35.726554 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.726286    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:36:36.226259549 +0000 UTC m=+35.494293175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:35.726554 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:35.726354    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:36.226335584 +0000 UTC m=+35.494369209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:36:35.726554 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.726490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb403fd-1ca8-4352-a7c7-ff62805c099c-tmp-dir\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.726725 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.726565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-trusted-ca\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.726781 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.726756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-certificates\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.730932 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.730900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-installation-pull-secrets\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.731087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.730900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-image-registry-private-configuration\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.735860 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.735828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-bound-sa-token\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.735860 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.735843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzcft\" (UniqueName: \"kubernetes.io/projected/4cb403fd-1ca8-4352-a7c7-ff62805c099c-kube-api-access-vzcft\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:35.736043 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.736018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvlsf\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-kube-api-access-kvlsf\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:35.748398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.748360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:35.826297 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.826258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfkp\" (UniqueName: \"kubernetes.io/projected/7dd1fe03-285d-4060-9a10-10e9c46b5566-kube-api-access-6sfkp\") pod \"ingress-canary-hrs62\" (UID: \"7dd1fe03-285d-4060-9a10-10e9c46b5566\") " pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:35.826500 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.826332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd1fe03-285d-4060-9a10-10e9c46b5566-cert\") pod \"ingress-canary-hrs62\" (UID: \"7dd1fe03-285d-4060-9a10-10e9c46b5566\") " pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:35.829558 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.829527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dd1fe03-285d-4060-9a10-10e9c46b5566-cert\") pod \"ingress-canary-hrs62\" (UID: \"7dd1fe03-285d-4060-9a10-10e9c46b5566\") " pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:35.835679 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.835647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfkp\" (UniqueName: \"kubernetes.io/projected/7dd1fe03-285d-4060-9a10-10e9c46b5566-kube-api-access-6sfkp\") pod \"ingress-canary-hrs62\" (UID: \"7dd1fe03-285d-4060-9a10-10e9c46b5566\") " pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:35.871112 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:35.871056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hrs62"
Sep 12 02:36:36.230404 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.230363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:36.230404 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.230412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.230471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:36.230544    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:36.230590    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:36.230594    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:36.230613    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:36.230629    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:37.23060789 +0000 UTC m=+36.498641519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:36.230666    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:36:37.230640944 +0000 UTC m=+36.498674572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:36.230712 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:36.230682    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:37.230674098 +0000 UTC m=+36.498707723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:36:36.243420 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.243386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:36:36.243420 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.243406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:36:36.243672 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.243643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:36:36.246295 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.246271    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 02:36:36.246295 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.246287    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7mt5x\""
Sep 12 02:36:36.246493 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.246312    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9jkq\""
Sep 12 02:36:36.246493 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.246271    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 02:36:36.246780 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.246759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 02:36:36.247193 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:36.247173    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 02:36:37.239740 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:37.239557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:37.239751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:37.239757    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:37.239799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:37.239839    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:39.239816524 +0000 UTC m=+38.507850166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:37.239887    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:37.239892    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:37.239911    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:37.239944    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:36:39.239927102 +0000 UTC m=+38.507960744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:37.240148 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:37.239964    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:39.239951064 +0000 UTC m=+38.507984703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:36:37.444315 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:37.444273    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hrs62"]
Sep 12 02:36:37.455188 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:36:37.455161    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd1fe03_285d_4060_9a10_10e9c46b5566.slice/crio-558803b8d5d7622191029fce5639def00b93017e8a36de1f3c4313bd016ad841 WatchSource:0}: Error finding container 558803b8d5d7622191029fce5639def00b93017e8a36de1f3c4313bd016ad841: Status 404 returned error can't find the container with id 558803b8d5d7622191029fce5639def00b93017e8a36de1f3c4313bd016ad841
Sep 12 02:36:37.526181 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:37.526132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hrs62" event={"ID":"7dd1fe03-285d-4060-9a10-10e9c46b5566","Type":"ContainerStarted","Data":"558803b8d5d7622191029fce5639def00b93017e8a36de1f3c4313bd016ad841"}
Sep 12 02:36:37.529175 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:37.529142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerStarted","Data":"2e657c6fbea234e8bcec82c6a09249467d3577cf1781f77c977e95a1cdb87a47"}
Sep 12 02:36:38.535964 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:38.535924    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c7b4b6-0c5e-41af-9e15-d5736881babe" containerID="2e657c6fbea234e8bcec82c6a09249467d3577cf1781f77c977e95a1cdb87a47" exitCode=0
Sep 12 02:36:38.536800 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:38.536003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerDied","Data":"2e657c6fbea234e8bcec82c6a09249467d3577cf1781f77c977e95a1cdb87a47"}
Sep 12 02:36:39.257630 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:39.257590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:39.257838 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:39.257654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:39.257838 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:39.257725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:39.257838 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:39.257803    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:36:39.258004 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:39.257812    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:39.258004 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:39.257899    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:36:39.258004 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:39.257917    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:36:39.258004 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:39.257887    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:43.257864547 +0000 UTC m=+42.525898176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:36:39.258004 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:39.257950    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:36:43.257931633 +0000 UTC m=+42.525965258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:39.258004 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:39.257965    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:43.257958696 +0000 UTC m=+42.525992321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:36:39.543483 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:39.543384    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c7b4b6-0c5e-41af-9e15-d5736881babe" containerID="2b2ae34b7a615f799edfa60d3a337cc8f65b8bf04381bede688352f56890455c" exitCode=0
Sep 12 02:36:39.543483 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:39.543469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerDied","Data":"2b2ae34b7a615f799edfa60d3a337cc8f65b8bf04381bede688352f56890455c"}
Sep 12 02:36:40.550075 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:40.549982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7sp86" event={"ID":"11c7b4b6-0c5e-41af-9e15-d5736881babe","Type":"ContainerStarted","Data":"b6bde820d88974f0a9d8de1f71aae78e98c1889697f359783bef2c43c774539d"}
Sep 12 02:36:40.569549 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:40.569500    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7sp86" podStartSLOduration=10.163916317 podStartE2EDuration="39.569481692s" totalImagesPullingTime="29.405565375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:40.568333928 +0000 UTC m=+39.836367597" watchObservedRunningTime="2026-09-12 02:36:40.569481692 +0000 UTC m=+39.837515371"
Sep 12 02:36:40.687679 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:40.687646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:40.736891 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:40.736861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:40.863952 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:40.863837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:40.964406 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:40.964361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:41.557034 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:41.556990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hrs62" event={"ID":"7dd1fe03-285d-4060-9a10-10e9c46b5566","Type":"ContainerStarted","Data":"b7e7b373ff9fbb24e20ab8555f1cae6ff4ee5c7f588d856a857b0f8c43b78fe5"}
Sep 12 02:36:41.571075 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:41.570945    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hrs62" podStartSLOduration=3.3734317 podStartE2EDuration="6.570925649s" totalImagesPullingTime="3.197493949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:41.570786978 +0000 UTC m=+40.838820625" watchObservedRunningTime="2026-09-12 02:36:41.570925649 +0000 UTC m=+40.838959299"
Sep 12 02:36:42.142727 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:42.142697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:42.243682 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:42.243655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:43.289568 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:43.289522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:43.289608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:43.289628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:43.289682    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:43.289703    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:43.289743    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:43.289750    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:43.289759    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:51.289743023 +0000 UTC m=+50.557776648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:43.289789    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:51.289776924 +0000 UTC m=+50.557810549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:36:43.289943 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:43.289803    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:36:51.289797193 +0000 UTC m=+50.557830818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:51.348923 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:51.348871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:36:51.348923 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:51.348924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:36:51.348976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:51.349032    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:51.349116    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:51.349119    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:51.349138    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:51.349129    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:07.349111435 +0000 UTC m=+66.617145072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:51.349187    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:07.349171043 +0000 UTC m=+66.617204671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:36:51.349471 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:36:51.349199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:37:07.349193731 +0000 UTC m=+66.617227356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:37:01.518179 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:01.518144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grws5"
Sep 12 02:37:06.962733 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:06.962686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:37:06.965774 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:06.965754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 02:37:06.976938 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:06.976907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff92f6c9-f9cb-4d8d-9056-c160e2de66cc-original-pull-secret\") pod \"global-pull-secret-syncer-kw4fm\" (UID: \"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc\") " pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:37:07.063813 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.063758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:37:07.066517 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.066495    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 02:37:07.074591 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.074562    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 02:37:07.074650 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.074642    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:38:11.074623854 +0000 UTC m=+130.342657479 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : secret "metrics-daemon-secret" not found
Sep 12 02:37:07.164373 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.164327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:37:07.167013 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.166991    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 02:37:07.170116 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.170078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kw4fm"
Sep 12 02:37:07.176804 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.176786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 02:37:07.187672 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.187636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzlc6\" (UniqueName: \"kubernetes.io/projected/ad36f8e0-62b5-440b-adfd-a102899cac72-kube-api-access-gzlc6\") pod \"network-check-target-lslfl\" (UID: \"ad36f8e0-62b5-440b-adfd-a102899cac72\") " pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:37:07.365696 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.365635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:37:07.365696 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.365697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:37:07.365918 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.365750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:37:07.365918 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.365809    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:37:07.365918 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.365857    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:37:07.365918 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.365871    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:37:07.365918 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.365874    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:37:07.365918 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.365887    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:39.365865954 +0000 UTC m=+98.633899584 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:37:07.365918 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.365913    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:39.365903178 +0000 UTC m=+98.633936807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:37:07.366168 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:07.365928    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:37:39.365920703 +0000 UTC m=+98.633954328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:37:07.440759 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.440722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kw4fm"]
Sep 12 02:37:07.444192 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:37:07.444164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff92f6c9_f9cb_4d8d_9056_c160e2de66cc.slice/crio-b50cdec3fcb851cd9e094d9fe040f0ef90f590f373745dced820ee75232b558c WatchSource:0}: Error finding container b50cdec3fcb851cd9e094d9fe040f0ef90f590f373745dced820ee75232b558c: Status 404 returned error can't find the container with id b50cdec3fcb851cd9e094d9fe040f0ef90f590f373745dced820ee75232b558c
Sep 12 02:37:07.465622 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.465583    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7mt5x\""
Sep 12 02:37:07.474578 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.474484    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:37:07.618018 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.617982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kw4fm" event={"ID":"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc","Type":"ContainerStarted","Data":"b50cdec3fcb851cd9e094d9fe040f0ef90f590f373745dced820ee75232b558c"}
Sep 12 02:37:07.745150 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:07.744913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lslfl"]
Sep 12 02:37:07.747479 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:37:07.747451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad36f8e0_62b5_440b_adfd_a102899cac72.slice/crio-ba21031d2d8af2cd73b8f77ded82c68fd1165125857c3512b3eff4e053dea88c WatchSource:0}: Error finding container ba21031d2d8af2cd73b8f77ded82c68fd1165125857c3512b3eff4e053dea88c: Status 404 returned error can't find the container with id ba21031d2d8af2cd73b8f77ded82c68fd1165125857c3512b3eff4e053dea88c
Sep 12 02:37:08.624570 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:08.624531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lslfl" event={"ID":"ad36f8e0-62b5-440b-adfd-a102899cac72","Type":"ContainerStarted","Data":"ba21031d2d8af2cd73b8f77ded82c68fd1165125857c3512b3eff4e053dea88c"}
Sep 12 02:37:13.639564 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:13.639521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kw4fm" event={"ID":"ff92f6c9-f9cb-4d8d-9056-c160e2de66cc","Type":"ContainerStarted","Data":"01e30202167deb79fc37339358f65d0f6a1d6c894da0dbdf63574c0c7381e6f3"}
Sep 12 02:37:13.641143 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:13.641115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lslfl" event={"ID":"ad36f8e0-62b5-440b-adfd-a102899cac72","Type":"ContainerStarted","Data":"e1d1509ba2143bec797155f45a81df64e9e983509a246d75d567329ebc0795da"}
Sep 12 02:37:13.641271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:13.641254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:37:13.653526 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:13.653471    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kw4fm" podStartSLOduration=67.212309781 podStartE2EDuration="1m12.653456544s" totalImagesPullingTime="5.441146763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:13.652804223 +0000 UTC m=+72.920837870" watchObservedRunningTime="2026-09-12 02:37:13.653456544 +0000 UTC m=+72.921490192"
Sep 12 02:37:13.666450 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:13.666389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lslfl" podStartSLOduration=67.541662217 podStartE2EDuration="1m12.666367558s" totalImagesPullingTime="5.124705341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:13.66601023 +0000 UTC m=+72.934043879" watchObservedRunningTime="2026-09-12 02:37:13.666367558 +0000 UTC m=+72.934401206"
Sep 12 02:37:39.411242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:39.411055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:37:39.411242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:39.411179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:37:39.411242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:39.411208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:37:39.411242 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:39.411213    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:37:39.411242 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:39.411232    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:37:39.411863 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:39.411319    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:37:39.411863 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:39.411354    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:38:43.411334128 +0000 UTC m=+162.679367767 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:37:39.411863 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:39.411381    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:38:43.411367287 +0000 UTC m=+162.679400928 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:37:39.411863 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:39.411322    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:37:39.411863 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:37:39.411409    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:38:43.411402201 +0000 UTC m=+162.679435855 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:37:44.646579 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:37:44.646546    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lslfl"
Sep 12 02:38:11.147922 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:11.147871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:38:11.148449 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:11.148028    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 02:38:11.148449 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:11.148126    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:40:13.148085594 +0000 UTC m=+252.416119218 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : secret "metrics-daemon-secret" not found
Sep 12 02:38:38.546078 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:38.546025    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24"
Sep 12 02:38:38.552244 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:38.552196    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6" podUID="eee61980-927f-493a-8e93-fafe1447d816"
Sep 12 02:38:38.562403 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:38.562371    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6znn9" podUID="4cb403fd-1ca8-4352-a7c7-ff62805c099c"
Sep 12 02:38:38.850170 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:38.850083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6znn9"
Sep 12 02:38:38.850287 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:38.850083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:38:38.850287 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:38.850083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:38:39.256218 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:39.256180    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:38:43.486430 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:43.486382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:38:43.486430 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:43.486429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:43.486570    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:43.486598    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:43.486650    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls podName:4cb403fd-1ca8-4352-a7c7-ff62805c099c nodeName:}" failed. No retries permitted until 2026-09-12 02:40:45.486634686 +0000 UTC m=+284.754668310 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls") pod "dns-default-6znn9" (UID: "4cb403fd-1ca8-4352-a7c7-ff62805c099c") : secret "dns-default-metrics-tls" not found
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:43.486664    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert podName:eee61980-927f-493a-8e93-fafe1447d816 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:45.486658057 +0000 UTC m=+284.754691683 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-tm7d6" (UID: "eee61980-927f-493a-8e93-fafe1447d816") : secret "networking-console-plugin-cert" not found
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:43.486645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:43.486718    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:43.486734    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85d54dd6dd-xkg4p: secret "image-registry-tls" not found
Sep 12 02:38:43.487134 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:38:43.486775    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls podName:56373d50-df04-4236-80c5-9de7b5c6dc24 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:45.4867623 +0000 UTC m=+284.754795925 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls") pod "image-registry-85d54dd6dd-xkg4p" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24") : secret "image-registry-tls" not found
Sep 12 02:38:52.243426 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:38:52.243345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:40:03.443837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.443805    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"]
Sep 12 02:40:03.446592 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.446576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.448880 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.448847    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 12 02:40:03.449056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.448974    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 12 02:40:03.450058 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.450037    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 12 02:40:03.450461 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.450432    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:40:03.450597 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.450565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pv2b5\""
Sep 12 02:40:03.451603 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.451585    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"]
Sep 12 02:40:03.454182 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.454165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"]
Sep 12 02:40:03.454308 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.454291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"
Sep 12 02:40:03.456899 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.456879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-66kx2\""
Sep 12 02:40:03.456988 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.456905    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-57tmp"]
Sep 12 02:40:03.457071 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.456995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"
Sep 12 02:40:03.459325 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.459308    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 12 02:40:03.459586 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.459564    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"]
Sep 12 02:40:03.459680 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.459630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.459680 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.459661    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:40:03.459855 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.459824    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7rv4n\""
Sep 12 02:40:03.461043 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.461020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-serving-cert\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.461146 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.461052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-config\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.461211 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.461149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fpw\" (UniqueName: \"kubernetes.io/projected/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-kube-api-access-z2fpw\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.462065 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.462035    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 12 02:40:03.462065 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.462056    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 12 02:40:03.462539 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.462074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zpxv9\""
Sep 12 02:40:03.462539 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.462044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 12 02:40:03.462817 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.462790    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 12 02:40:03.470263 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.469958    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"]
Sep 12 02:40:03.470989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.470965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"]
Sep 12 02:40:03.472053 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.472029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-57tmp"]
Sep 12 02:40:03.472575 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.472555    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 12 02:40:03.549243 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.549210    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"]
Sep 12 02:40:03.552210 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.552194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.554688 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.554668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 12 02:40:03.555224 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.555195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 12 02:40:03.555317 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.555228    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 12 02:40:03.555317 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.555253    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-z6zmb\""
Sep 12 02:40:03.555317 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.555256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:40:03.562245 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-serving-cert\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.562334 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-config\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.562334 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88h6l\" (UniqueName: \"kubernetes.io/projected/955fa226-c61a-42f7-816c-cb8d6a02e748-kube-api-access-88h6l\") pod \"network-check-source-6f8b768455-bgkf2\" (UID: \"955fa226-c61a-42f7-816c-cb8d6a02e748\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"
Sep 12 02:40:03.562334 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4344e4db-5901-4556-916e-c38e0ca4f7a6-snapshots\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.562334 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4344e4db-5901-4556-916e-c38e0ca4f7a6-serving-cert\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.562334 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4344e4db-5901-4556-916e-c38e0ca4f7a6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.562594 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nb94\" (UniqueName: \"kubernetes.io/projected/4344e4db-5901-4556-916e-c38e0ca4f7a6-kube-api-access-2nb94\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.562594 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4344e4db-5901-4556-916e-c38e0ca4f7a6-tmp\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.562594 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4344e4db-5901-4556-916e-c38e0ca4f7a6-service-ca-bundle\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.562594 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fpw\" (UniqueName: \"kubernetes.io/projected/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-kube-api-access-z2fpw\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.562594 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49kv\" (UniqueName: \"kubernetes.io/projected/3e9b3ae8-9baa-4efb-b8de-01b627a7505a-kube-api-access-j49kv\") pod \"volume-data-source-validator-86695b7cc7-xqh5z\" (UID: \"3e9b3ae8-9baa-4efb-b8de-01b627a7505a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"
Sep 12 02:40:03.562791 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.562738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-config\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.564129 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.564082    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"]
Sep 12 02:40:03.565021 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.564998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-serving-cert\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.578293 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.578266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fpw\" (UniqueName: \"kubernetes.io/projected/7fdbe5e6-27c1-4170-9ba4-e0ec798cb927-kube-api-access-z2fpw\") pod \"service-ca-operator-cf7c776b7-dnfww\" (UID: \"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.663644 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4344e4db-5901-4556-916e-c38e0ca4f7a6-snapshots\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.663644 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fd11876-ba7c-415c-8b18-c450d94fa012-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.663874 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fd11876-ba7c-415c-8b18-c450d94fa012-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.663874 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49kv\" (UniqueName: \"kubernetes.io/projected/3e9b3ae8-9baa-4efb-b8de-01b627a7505a-kube-api-access-j49kv\") pod \"volume-data-source-validator-86695b7cc7-xqh5z\" (UID: \"3e9b3ae8-9baa-4efb-b8de-01b627a7505a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"
Sep 12 02:40:03.663874 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nb94\" (UniqueName: \"kubernetes.io/projected/4344e4db-5901-4556-916e-c38e0ca4f7a6-kube-api-access-2nb94\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.663996 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4344e4db-5901-4556-916e-c38e0ca4f7a6-serving-cert\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.663996 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4344e4db-5901-4556-916e-c38e0ca4f7a6-tmp\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.663996 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4344e4db-5901-4556-916e-c38e0ca4f7a6-service-ca-bundle\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.663996 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.663992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9r88\" (UniqueName: \"kubernetes.io/projected/9fd11876-ba7c-415c-8b18-c450d94fa012-kube-api-access-f9r88\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.664225 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.664020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88h6l\" (UniqueName: \"kubernetes.io/projected/955fa226-c61a-42f7-816c-cb8d6a02e748-kube-api-access-88h6l\") pod \"network-check-source-6f8b768455-bgkf2\" (UID: \"955fa226-c61a-42f7-816c-cb8d6a02e748\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"
Sep 12 02:40:03.664225 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.664050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4344e4db-5901-4556-916e-c38e0ca4f7a6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.664342 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.664322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4344e4db-5901-4556-916e-c38e0ca4f7a6-tmp\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.664445 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.664427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4344e4db-5901-4556-916e-c38e0ca4f7a6-snapshots\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.664561 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.664545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4344e4db-5901-4556-916e-c38e0ca4f7a6-service-ca-bundle\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.664790 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.664774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4344e4db-5901-4556-916e-c38e0ca4f7a6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.666362 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.666347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4344e4db-5901-4556-916e-c38e0ca4f7a6-serving-cert\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.676606 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.676572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nb94\" (UniqueName: \"kubernetes.io/projected/4344e4db-5901-4556-916e-c38e0ca4f7a6-kube-api-access-2nb94\") pod \"insights-operator-5844b856db-57tmp\" (UID: \"4344e4db-5901-4556-916e-c38e0ca4f7a6\") " pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.676606 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.676591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49kv\" (UniqueName: \"kubernetes.io/projected/3e9b3ae8-9baa-4efb-b8de-01b627a7505a-kube-api-access-j49kv\") pod \"volume-data-source-validator-86695b7cc7-xqh5z\" (UID: \"3e9b3ae8-9baa-4efb-b8de-01b627a7505a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"
Sep 12 02:40:03.676742 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.676648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88h6l\" (UniqueName: \"kubernetes.io/projected/955fa226-c61a-42f7-816c-cb8d6a02e748-kube-api-access-88h6l\") pod \"network-check-source-6f8b768455-bgkf2\" (UID: \"955fa226-c61a-42f7-816c-cb8d6a02e748\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"
Sep 12 02:40:03.756351 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.756265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"
Sep 12 02:40:03.764530 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.764512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fd11876-ba7c-415c-8b18-c450d94fa012-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.764609 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.764538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fd11876-ba7c-415c-8b18-c450d94fa012-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.764668 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.764606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9r88\" (UniqueName: \"kubernetes.io/projected/9fd11876-ba7c-415c-8b18-c450d94fa012-kube-api-access-f9r88\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.765386 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.765358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fd11876-ba7c-415c-8b18-c450d94fa012-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.766848 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.766815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fd11876-ba7c-415c-8b18-c450d94fa012-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.766929 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.766902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"
Sep 12 02:40:03.772426 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.772409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9r88\" (UniqueName: \"kubernetes.io/projected/9fd11876-ba7c-415c-8b18-c450d94fa012-kube-api-access-f9r88\") pod \"kube-storage-version-migrator-operator-6c8f84f897-brgwx\" (UID: \"9fd11876-ba7c-415c-8b18-c450d94fa012\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:03.776331 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.776310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"
Sep 12 02:40:03.781196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.781177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-57tmp"
Sep 12 02:40:03.860591 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:03.860567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"
Sep 12 02:40:04.035191 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:04.035071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww"]
Sep 12 02:40:04.037556 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:04.037526    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdbe5e6_27c1_4170_9ba4_e0ec798cb927.slice/crio-06f27805c3f78bac0fb60880f756722e07261e135a144ad2c4b03dfa1b1db15e WatchSource:0}: Error finding container 06f27805c3f78bac0fb60880f756722e07261e135a144ad2c4b03dfa1b1db15e: Status 404 returned error can't find the container with id 06f27805c3f78bac0fb60880f756722e07261e135a144ad2c4b03dfa1b1db15e
Sep 12 02:40:04.059414 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:04.059381    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2"]
Sep 12 02:40:04.059818 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:04.059788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww" event={"ID":"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927","Type":"ContainerStarted","Data":"06f27805c3f78bac0fb60880f756722e07261e135a144ad2c4b03dfa1b1db15e"}
Sep 12 02:40:04.061703 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:04.061680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z"]
Sep 12 02:40:04.062025 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:04.062003    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955fa226_c61a_42f7_816c_cb8d6a02e748.slice/crio-851ea53669350483f5ac771ff5244273d2ea67165cb7f8b4775f92bd705ef014 WatchSource:0}: Error finding container 851ea53669350483f5ac771ff5244273d2ea67165cb7f8b4775f92bd705ef014: Status 404 returned error can't find the container with id 851ea53669350483f5ac771ff5244273d2ea67165cb7f8b4775f92bd705ef014
Sep 12 02:40:04.064774 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:04.064683    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9b3ae8_9baa_4efb_b8de_01b627a7505a.slice/crio-98a9a2a96a7c4a34e60473466a3679e9f06f2453e5d9e262b749dde0c4154c8b WatchSource:0}: Error finding container 98a9a2a96a7c4a34e60473466a3679e9f06f2453e5d9e262b749dde0c4154c8b: Status 404 returned error can't find the container with id 98a9a2a96a7c4a34e60473466a3679e9f06f2453e5d9e262b749dde0c4154c8b
Sep 12 02:40:04.064995 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:04.064976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-57tmp"]
Sep 12 02:40:04.067687 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:04.067667    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4344e4db_5901_4556_916e_c38e0ca4f7a6.slice/crio-96bd3fdb361fb5bc8b2af86b880f2d5518328305de095a182920bcc2f0b7c493 WatchSource:0}: Error finding container 96bd3fdb361fb5bc8b2af86b880f2d5518328305de095a182920bcc2f0b7c493: Status 404 returned error can't find the container with id 96bd3fdb361fb5bc8b2af86b880f2d5518328305de095a182920bcc2f0b7c493
Sep 12 02:40:04.133260 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:04.133233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx"]
Sep 12 02:40:04.136536 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:04.136490    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd11876_ba7c_415c_8b18_c450d94fa012.slice/crio-56e8636ded270e137b9ce9e0f57a72ef63057f5382e69b861a318bab6ab25111 WatchSource:0}: Error finding container 56e8636ded270e137b9ce9e0f57a72ef63057f5382e69b861a318bab6ab25111: Status 404 returned error can't find the container with id 56e8636ded270e137b9ce9e0f57a72ef63057f5382e69b861a318bab6ab25111
Sep 12 02:40:05.066389 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:05.065187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx" event={"ID":"9fd11876-ba7c-415c-8b18-c450d94fa012","Type":"ContainerStarted","Data":"56e8636ded270e137b9ce9e0f57a72ef63057f5382e69b861a318bab6ab25111"}
Sep 12 02:40:05.067140 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:05.067061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-57tmp" event={"ID":"4344e4db-5901-4556-916e-c38e0ca4f7a6","Type":"ContainerStarted","Data":"96bd3fdb361fb5bc8b2af86b880f2d5518328305de095a182920bcc2f0b7c493"}
Sep 12 02:40:05.068714 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:05.068646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z" event={"ID":"3e9b3ae8-9baa-4efb-b8de-01b627a7505a","Type":"ContainerStarted","Data":"98a9a2a96a7c4a34e60473466a3679e9f06f2453e5d9e262b749dde0c4154c8b"}
Sep 12 02:40:05.072819 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:05.072290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2" event={"ID":"955fa226-c61a-42f7-816c-cb8d6a02e748","Type":"ContainerStarted","Data":"546753c5767cca8f60cc841b781e05cde5036d4c340cce11800a6bc0e1816484"}
Sep 12 02:40:05.072819 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:05.072335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2" event={"ID":"955fa226-c61a-42f7-816c-cb8d6a02e748","Type":"ContainerStarted","Data":"851ea53669350483f5ac771ff5244273d2ea67165cb7f8b4775f92bd705ef014"}
Sep 12 02:40:05.089794 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:05.089068    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bgkf2" podStartSLOduration=2.089050565 podStartE2EDuration="2.089050565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:05.088312616 +0000 UTC m=+244.356346265" watchObservedRunningTime="2026-09-12 02:40:05.089050565 +0000 UTC m=+244.357084211"
Sep 12 02:40:06.079087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:06.079048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z" event={"ID":"3e9b3ae8-9baa-4efb-b8de-01b627a7505a","Type":"ContainerStarted","Data":"989eb5d3c36e929b0e47b5cae1b811a35113b96161f5daa0be712a951a15e222"}
Sep 12 02:40:06.093035 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:06.092983    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xqh5z" podStartSLOduration=1.7155090899999998 podStartE2EDuration="3.092965865s" totalImagesPullingTime="1.377456775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:06.092424015 +0000 UTC m=+245.360457679" watchObservedRunningTime="2026-09-12 02:40:06.092965865 +0000 UTC m=+245.360999513"
Sep 12 02:40:08.090416 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.090386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww" event={"ID":"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927","Type":"ContainerStarted","Data":"bc964729d1c76fda887de5ddabd8cacb9eed084feada335551d5816979504f2b"}
Sep 12 02:40:08.092452 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.092241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx" event={"ID":"9fd11876-ba7c-415c-8b18-c450d94fa012","Type":"ContainerStarted","Data":"4459f5fb6adb6f50c2fa4bc1aed94f3c14d5228b6c39f5a7ea1b6bb342b4151b"}
Sep 12 02:40:08.094464 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.094440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-57tmp" event={"ID":"4344e4db-5901-4556-916e-c38e0ca4f7a6","Type":"ContainerStarted","Data":"d3ced2e58324538080e9d2e1557488cd732494451cd5ac8d556efdd1e019643f"}
Sep 12 02:40:08.101868 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.101840    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9crmx"]
Sep 12 02:40:08.105261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.105241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.107420 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.107371    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww" podStartSLOduration=1.7940531499999999 podStartE2EDuration="5.107352848s" totalImagesPullingTime="3.313299698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:08.106739086 +0000 UTC m=+247.374772723" watchObservedRunningTime="2026-09-12 02:40:08.107352848 +0000 UTC m=+247.375386496"
Sep 12 02:40:08.107542 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.107470    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 12 02:40:08.107542 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.107529    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 12 02:40:08.107728 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.107709    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-klkm9\""
Sep 12 02:40:08.126441 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.126406    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9crmx"]
Sep 12 02:40:08.146400 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.146336    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-57tmp" podStartSLOduration=1.867281617 podStartE2EDuration="5.146314531s" totalImagesPullingTime="3.279032914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:08.146306198 +0000 UTC m=+247.414339855" watchObservedRunningTime="2026-09-12 02:40:08.146314531 +0000 UTC m=+247.414348171"
Sep 12 02:40:08.176411 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.176355    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx" podStartSLOduration=1.969404207 podStartE2EDuration="5.176339252s" totalImagesPullingTime="3.206935045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:08.175587094 +0000 UTC m=+247.443620741" watchObservedRunningTime="2026-09-12 02:40:08.176339252 +0000 UTC m=+247.444372898"
Sep 12 02:40:08.200710 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.200672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.200710 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.200703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.200932 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.200893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40cb9ec2-d376-4703-a245-ca7c5cd87443-data-volume\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.200991 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.200947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40cb9ec2-d376-4703-a245-ca7c5cd87443-crio-socket\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.200991 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.200977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnfgf\" (UniqueName: \"kubernetes.io/projected/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-api-access-wnfgf\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.302363 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.302304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.302363 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.302361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.302616 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.302433    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.302616 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.302463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40cb9ec2-d376-4703-a245-ca7c5cd87443-data-volume\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.302616 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.302503    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:08.802486777 +0000 UTC m=+248.070520405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.302616 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.302520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40cb9ec2-d376-4703-a245-ca7c5cd87443-crio-socket\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.302616 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.302553    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:08.302616 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.302562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnfgf\" (UniqueName: \"kubernetes.io/projected/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-api-access-wnfgf\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.302616 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.302612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:08.802596227 +0000 UTC m=+248.070629867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:08.302921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.302631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/40cb9ec2-d376-4703-a245-ca7c5cd87443-crio-socket\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.302921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.302807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/40cb9ec2-d376-4703-a245-ca7c5cd87443-data-volume\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.312127 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.312080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnfgf\" (UniqueName: \"kubernetes.io/projected/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-api-access-wnfgf\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.806795 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.806742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.806795 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:08.806797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:08.807032 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.806868    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.807032 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.806906    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:08.807032 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.806937    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:09.80692173 +0000 UTC m=+249.074955355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.807032 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:08.806970    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:09.806954081 +0000 UTC m=+249.074987707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:09.568878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.568842    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"]
Sep 12 02:40:09.574901 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.574873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"
Sep 12 02:40:09.577375 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.577346    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7tbjs\""
Sep 12 02:40:09.577507 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.577376    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 12 02:40:09.577507 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.577401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 12 02:40:09.581290 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.581267    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"]
Sep 12 02:40:09.613333 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.613304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wmw\" (UniqueName: \"kubernetes.io/projected/c7f29df3-0ad6-475c-b3ed-a60fb4824f5d-kube-api-access-t6wmw\") pod \"migrator-8b4d697c5-s65df\" (UID: \"c7f29df3-0ad6-475c-b3ed-a60fb4824f5d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"
Sep 12 02:40:09.714036 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.713998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wmw\" (UniqueName: \"kubernetes.io/projected/c7f29df3-0ad6-475c-b3ed-a60fb4824f5d-kube-api-access-t6wmw\") pod \"migrator-8b4d697c5-s65df\" (UID: \"c7f29df3-0ad6-475c-b3ed-a60fb4824f5d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"
Sep 12 02:40:09.722050 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.722020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wmw\" (UniqueName: \"kubernetes.io/projected/c7f29df3-0ad6-475c-b3ed-a60fb4824f5d-kube-api-access-t6wmw\") pod \"migrator-8b4d697c5-s65df\" (UID: \"c7f29df3-0ad6-475c-b3ed-a60fb4824f5d\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"
Sep 12 02:40:09.814451 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.814413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:09.814451 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.814453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:09.814705 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:09.814531    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:09.814705 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:09.814597    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:11.814581114 +0000 UTC m=+251.082614739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:09.814705 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:09.814649    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:09.814832 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:09.814727    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:11.814709193 +0000 UTC m=+251.082742836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:09.872418 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.872345    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7flt6_b0f02bda-791d-4340-a758-2269e6c97bf4/dns-node-resolver/0.log"
Sep 12 02:40:09.885193 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:09.885162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"
Sep 12 02:40:10.150875 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.150799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df"]
Sep 12 02:40:10.153615 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:10.153585    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f29df3_0ad6_475c_b3ed_a60fb4824f5d.slice/crio-1283281d0056ee44d39e248486f9833b925163da1e5692439a85949214fa9804 WatchSource:0}: Error finding container 1283281d0056ee44d39e248486f9833b925163da1e5692439a85949214fa9804: Status 404 returned error can't find the container with id 1283281d0056ee44d39e248486f9833b925163da1e5692439a85949214fa9804
Sep 12 02:40:10.874149 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.874116    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-85lmg_3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de/node-ca/0.log"
Sep 12 02:40:10.874771 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.874744    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tqbts"]
Sep 12 02:40:10.878162 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.878144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:10.880613 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.880571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 12 02:40:10.881989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.881779    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 12 02:40:10.881989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.881798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rxflz\""
Sep 12 02:40:10.881989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.881825    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 12 02:40:10.881989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.881846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 12 02:40:10.886266 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.886246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tqbts"]
Sep 12 02:40:10.922731 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.922699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-signing-cabundle\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:10.922900 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.922768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhvr\" (UniqueName: \"kubernetes.io/projected/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-kube-api-access-snhvr\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:10.922900 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:10.922822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-signing-key\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.023819 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.023786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-signing-cabundle\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.023981 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.023839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhvr\" (UniqueName: \"kubernetes.io/projected/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-kube-api-access-snhvr\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.024028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.023982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-signing-key\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.024522 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.024503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-signing-cabundle\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.026472 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.026450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-signing-key\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.033013 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.032988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhvr\" (UniqueName: \"kubernetes.io/projected/780f9f9c-e5f7-4e61-9633-9e4af3b62df1-kube-api-access-snhvr\") pod \"service-ca-568869fdc5-tqbts\" (UID: \"780f9f9c-e5f7-4e61-9633-9e4af3b62df1\") " pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.105515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.105477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df" event={"ID":"c7f29df3-0ad6-475c-b3ed-a60fb4824f5d","Type":"ContainerStarted","Data":"1283281d0056ee44d39e248486f9833b925163da1e5692439a85949214fa9804"}
Sep 12 02:40:11.190179 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.190141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-tqbts"
Sep 12 02:40:11.627025 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.626988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tqbts"]
Sep 12 02:40:11.631467 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:11.631433    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780f9f9c_e5f7_4e61_9633_9e4af3b62df1.slice/crio-3d8b48adcaa6de9e71bc1f61aa071c893b6d4fce17f723ea4b5272a89bf7bb55 WatchSource:0}: Error finding container 3d8b48adcaa6de9e71bc1f61aa071c893b6d4fce17f723ea4b5272a89bf7bb55: Status 404 returned error can't find the container with id 3d8b48adcaa6de9e71bc1f61aa071c893b6d4fce17f723ea4b5272a89bf7bb55
Sep 12 02:40:11.672026 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.672006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hrs62_7dd1fe03-285d-4060-9a10-10e9c46b5566/serve-healthcheck-canary/0.log"
Sep 12 02:40:11.831677 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.831638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:11.831677 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:11.831675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:11.831924 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:11.831770    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:11.831924 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:11.831794    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:11.831924 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:11.831844    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:15.831830073 +0000 UTC m=+255.099863698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:11.831924 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:11.831859    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:15.83185299 +0000 UTC m=+255.099886615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:12.110308 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:12.110214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df" event={"ID":"c7f29df3-0ad6-475c-b3ed-a60fb4824f5d","Type":"ContainerStarted","Data":"30fba47050700194fd420c26d7db77ca07e955d0bdb0cce7061ca8d90d35e45d"}
Sep 12 02:40:12.110308 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:12.110276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df" event={"ID":"c7f29df3-0ad6-475c-b3ed-a60fb4824f5d","Type":"ContainerStarted","Data":"8d6cd9e553bb4e43abbed64a743a85683429c74022cabd9ec6b7c59c3c7cd7cd"}
Sep 12 02:40:12.111887 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:12.111864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-tqbts" event={"ID":"780f9f9c-e5f7-4e61-9633-9e4af3b62df1","Type":"ContainerStarted","Data":"062b6485c41a29d24fcc45f8ed2c237165c2e17229ebc0a951d30810560a3f6f"}
Sep 12 02:40:12.111966 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:12.111903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-tqbts" event={"ID":"780f9f9c-e5f7-4e61-9633-9e4af3b62df1","Type":"ContainerStarted","Data":"3d8b48adcaa6de9e71bc1f61aa071c893b6d4fce17f723ea4b5272a89bf7bb55"}
Sep 12 02:40:12.125077 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:12.125031    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s65df" podStartSLOduration=1.873364713 podStartE2EDuration="3.125017439s" totalImagesPullingTime="1.251652726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:12.123982258 +0000 UTC m=+251.392015904" watchObservedRunningTime="2026-09-12 02:40:12.125017439 +0000 UTC m=+251.393051085"
Sep 12 02:40:12.141480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:12.141433    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-tqbts" podStartSLOduration=2.141416823 podStartE2EDuration="2.141416823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:12.140625269 +0000 UTC m=+251.408658916" watchObservedRunningTime="2026-09-12 02:40:12.141416823 +0000 UTC m=+251.409450533"
Sep 12 02:40:13.242465 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:13.242421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:40:13.242956 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:13.242552    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 02:40:13.242956 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:13.242639    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs podName:c0eba23f-369a-4cec-954d-d152d68cc4bd nodeName:}" failed. No retries permitted until 2026-09-12 02:42:15.242618557 +0000 UTC m=+374.510652185 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs") pod "network-metrics-daemon-gncjn" (UID: "c0eba23f-369a-4cec-954d-d152d68cc4bd") : secret "metrics-daemon-secret" not found
Sep 12 02:40:15.865826 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:15.865787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:15.866223 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:15.865836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:15.866223 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:15.865900    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:15.866223 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:15.865974    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:23.865958066 +0000 UTC m=+263.133991691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:15.866223 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:15.865979    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:15.866223 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:15.866030    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls podName:40cb9ec2-d376-4703-a245-ca7c5cd87443 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:23.866014613 +0000 UTC m=+263.134048242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crmx" (UID: "40cb9ec2-d376-4703-a245-ca7c5cd87443") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:23.930554 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:23.930452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:23.930554 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:23.930507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:23.931052 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:23.930992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/40cb9ec2-d376-4703-a245-ca7c5cd87443-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:23.932939 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:23.932921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/40cb9ec2-d376-4703-a245-ca7c5cd87443-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crmx\" (UID: \"40cb9ec2-d376-4703-a245-ca7c5cd87443\") " pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:24.016417 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:24.016382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9crmx"
Sep 12 02:40:24.286365 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:24.286288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9crmx"]
Sep 12 02:40:24.289286 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:24.289258    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cb9ec2_d376_4703_a245_ca7c5cd87443.slice/crio-cbc3a5fd4fcf0d330b79ed6e717a91f5ac82a6efbc840a97f8f98eed142d76da WatchSource:0}: Error finding container cbc3a5fd4fcf0d330b79ed6e717a91f5ac82a6efbc840a97f8f98eed142d76da: Status 404 returned error can't find the container with id cbc3a5fd4fcf0d330b79ed6e717a91f5ac82a6efbc840a97f8f98eed142d76da
Sep 12 02:40:25.154241 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:25.154206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crmx" event={"ID":"40cb9ec2-d376-4703-a245-ca7c5cd87443","Type":"ContainerStarted","Data":"eef8ca932f92c42adde3602188f064ae6e6586a686f58b6cf57afd32b7f19419"}
Sep 12 02:40:25.154587 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:25.154263    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crmx" event={"ID":"40cb9ec2-d376-4703-a245-ca7c5cd87443","Type":"ContainerStarted","Data":"d073e5fa0edcc776d9f76b691aeb64ff8330efd42d7aa4f1f10393fb5aaf1139"}
Sep 12 02:40:25.154587 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:25.154280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crmx" event={"ID":"40cb9ec2-d376-4703-a245-ca7c5cd87443","Type":"ContainerStarted","Data":"cbc3a5fd4fcf0d330b79ed6e717a91f5ac82a6efbc840a97f8f98eed142d76da"}
Sep 12 02:40:27.165518 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:27.165478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crmx" event={"ID":"40cb9ec2-d376-4703-a245-ca7c5cd87443","Type":"ContainerStarted","Data":"dd3e7a06b4f0aa419772f75624cb232b981d637fa614395bcae77aa19493629a"}
Sep 12 02:40:27.180344 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:27.180300    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9crmx" podStartSLOduration=17.109532946 podStartE2EDuration="19.180286299s" totalImagesPullingTime="2.070753353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:40:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:27.17973933 +0000 UTC m=+266.447772976" watchObservedRunningTime="2026-09-12 02:40:27.180286299 +0000 UTC m=+266.448319945"
Sep 12 02:40:41.850903 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:41.850857    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24"
Sep 12 02:40:41.850903 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:41.850884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6" podUID="eee61980-927f-493a-8e93-fafe1447d816"
Sep 12 02:40:41.851365 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:41.850884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6znn9" podUID="4cb403fd-1ca8-4352-a7c7-ff62805c099c"
Sep 12 02:40:42.210878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:42.210851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:40:42.211042 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:42.210851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6znn9"
Sep 12 02:40:42.211042 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:42.210853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:40:45.447660 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.447621    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"]
Sep 12 02:40:45.451275 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.451182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.453482 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.453459    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 12 02:40:45.454968 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.454943    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zxxhq\""
Sep 12 02:40:45.454968 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.454960    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 12 02:40:45.455167 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.455015    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 12 02:40:45.455167 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.454952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 12 02:40:45.455374 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.455349    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 12 02:40:45.459983 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.459960    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"]
Sep 12 02:40:45.462158 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.462140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.464591 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.464572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 12 02:40:45.464765 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.464745    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 12 02:40:45.465078 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.465051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6fckc\""
Sep 12 02:40:45.465450 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.465432    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 12 02:40:45.466659 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.466638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"]
Sep 12 02:40:45.482252 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.482227    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zvqz6"]
Sep 12 02:40:45.488984 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.488961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.490956 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.490934    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"]
Sep 12 02:40:45.491601 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.491576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 12 02:40:45.491829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.491812    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gqzx9\""
Sep 12 02:40:45.491906 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.491828    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 12 02:40:45.492046 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.492027    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 12 02:40:45.499709 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.499687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:40:45.499849 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.499736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:40:45.499849 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.499792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:40:45.502490 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.502463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eee61980-927f-493a-8e93-fafe1447d816-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-tm7d6\" (UID: \"eee61980-927f-493a-8e93-fafe1447d816\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:40:45.502634 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.502616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cb403fd-1ca8-4352-a7c7-ff62805c099c-metrics-tls\") pod \"dns-default-6znn9\" (UID: \"4cb403fd-1ca8-4352-a7c7-ff62805c099c\") " pod="openshift-dns/dns-default-6znn9"
Sep 12 02:40:45.502855 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.502832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"image-registry-85d54dd6dd-xkg4p\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") " pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:40:45.513476 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.513451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-88r85\""
Sep 12 02:40:45.513613 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.513541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jtp6k\""
Sep 12 02:40:45.513692 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.513623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4279k\""
Sep 12 02:40:45.521785 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.521758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:40:45.521885 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.521815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6znn9"
Sep 12 02:40:45.521885 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.521762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"
Sep 12 02:40:45.600475 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269ee6aa-9c67-42a7-af13-617886fb4ad2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.600588 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-textfile\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.600588 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.600588 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-tls\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.600754 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269ee6aa-9c67-42a7-af13-617886fb4ad2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.600754 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-root\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.600754 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-wtmp\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.600754 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.600948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/269ee6aa-9c67-42a7-af13-617886fb4ad2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.600948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.600948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.600948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.600948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzct\" (UniqueName: \"kubernetes.io/projected/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-kube-api-access-wmzct\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.600948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-sys\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.601343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.601343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.600996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2kzd\" (UniqueName: \"kubernetes.io/projected/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-api-access-n2kzd\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.601343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.601027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.601343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.601052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-metrics-client-ca\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.601343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.601084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjsp6\" (UniqueName: \"kubernetes.io/projected/269ee6aa-9c67-42a7-af13-617886fb4ad2-kube-api-access-sjsp6\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.701891 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.701807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-sys\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.701891 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.701849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.701891 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.701880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2kzd\" (UniqueName: \"kubernetes.io/projected/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-api-access-n2kzd\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.701905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.701917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-sys\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.701930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-metrics-client-ca\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.701976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjsp6\" (UniqueName: \"kubernetes.io/projected/269ee6aa-9c67-42a7-af13-617886fb4ad2-kube-api-access-sjsp6\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269ee6aa-9c67-42a7-af13-617886fb4ad2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-textfile\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-tls\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702168 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269ee6aa-9c67-42a7-af13-617886fb4ad2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-root\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-wtmp\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/269ee6aa-9c67-42a7-af13-617886fb4ad2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.702605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzct\" (UniqueName: \"kubernetes.io/projected/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-kube-api-access-wmzct\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702983 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-metrics-client-ca\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702983 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-textfile\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.702983 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-wtmp\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.703583 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.703472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.703887 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.703646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269ee6aa-9c67-42a7-af13-617886fb4ad2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.703963 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.702502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-root\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.704604 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.704576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.704692 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:45.704677    2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 12 02:40:45.704758 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:45.704731    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-tls podName:c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:46.204714103 +0000 UTC m=+285.472747731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-mwwbr" (UID: "c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4") : secret "kube-state-metrics-tls" not found
Sep 12 02:40:45.705208 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.705186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.705559 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.705529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-tls\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.705702 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.705684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.705941 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.705915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269ee6aa-9c67-42a7-af13-617886fb4ad2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.706208 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.706185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.706747 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.706728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/269ee6aa-9c67-42a7-af13-617886fb4ad2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.708117 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.708072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.710488 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.710462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzct\" (UniqueName: \"kubernetes.io/projected/a0e5173a-6ced-49c0-8c95-0fdaecf85e42-kube-api-access-wmzct\") pod \"node-exporter-zvqz6\" (UID: \"a0e5173a-6ced-49c0-8c95-0fdaecf85e42\") " pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.710663 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.710641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2kzd\" (UniqueName: \"kubernetes.io/projected/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-api-access-n2kzd\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:45.710709 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.710678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjsp6\" (UniqueName: \"kubernetes.io/projected/269ee6aa-9c67-42a7-af13-617886fb4ad2-kube-api-access-sjsp6\") pod \"openshift-state-metrics-67dcf8c979-4td4t\" (UID: \"269ee6aa-9c67-42a7-af13-617886fb4ad2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.761528 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.761489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"
Sep 12 02:40:45.800867 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.800829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvqz6"
Sep 12 02:40:45.814082 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.813008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6znn9"]
Sep 12 02:40:45.816419 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:45.816378    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e5173a_6ced_49c0_8c95_0fdaecf85e42.slice/crio-e0dd822699737ec20986f6365433dd62bed8e64532542f3cb3a287ddda20ff36 WatchSource:0}: Error finding container e0dd822699737ec20986f6365433dd62bed8e64532542f3cb3a287ddda20ff36: Status 404 returned error can't find the container with id e0dd822699737ec20986f6365433dd62bed8e64532542f3cb3a287ddda20ff36
Sep 12 02:40:45.821454 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.820766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d54dd6dd-xkg4p"]
Sep 12 02:40:45.821454 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:45.820835    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6"]
Sep 12 02:40:45.826441 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:45.826414    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56373d50_df04_4236_80c5_9de7b5c6dc24.slice/crio-fb435327703c002d62d6a45d4432611066f050210042421024528e55a369aa64 WatchSource:0}: Error finding container fb435327703c002d62d6a45d4432611066f050210042421024528e55a369aa64: Status 404 returned error can't find the container with id fb435327703c002d62d6a45d4432611066f050210042421024528e55a369aa64
Sep 12 02:40:45.828211 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:45.827908    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee61980_927f_493a_8e93_fafe1447d816.slice/crio-6240a98cff704d4efc5dcd8fd32330aed1bda7496eec7fdae705d86686790235 WatchSource:0}: Error finding container 6240a98cff704d4efc5dcd8fd32330aed1bda7496eec7fdae705d86686790235: Status 404 returned error can't find the container with id 6240a98cff704d4efc5dcd8fd32330aed1bda7496eec7fdae705d86686790235
Sep 12 02:40:46.033554 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.033527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t"]
Sep 12 02:40:46.035631 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:46.035592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269ee6aa_9c67_42a7_af13_617886fb4ad2.slice/crio-2807e705d0c3d15fbf18a55d15369d8c548a319751fc41908377d418f984b230 WatchSource:0}: Error finding container 2807e705d0c3d15fbf18a55d15369d8c548a319751fc41908377d418f984b230: Status 404 returned error can't find the container with id 2807e705d0c3d15fbf18a55d15369d8c548a319751fc41908377d418f984b230
Sep 12 02:40:46.207111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.207051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:46.210075 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.210011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mwwbr\" (UID: \"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:46.227549 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.227445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t" event={"ID":"269ee6aa-9c67-42a7-af13-617886fb4ad2","Type":"ContainerStarted","Data":"bcc9cb163ca0b5e6b9025c33d7f4e636668043d531bb0964883d7f5fc4d7b88f"}
Sep 12 02:40:46.227549 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.227516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t" event={"ID":"269ee6aa-9c67-42a7-af13-617886fb4ad2","Type":"ContainerStarted","Data":"826183bea4bd3d06e21b4958d71fb289aadc36d6decc737e06a4a64906f2fb81"}
Sep 12 02:40:46.227549 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.227545    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t" event={"ID":"269ee6aa-9c67-42a7-af13-617886fb4ad2","Type":"ContainerStarted","Data":"2807e705d0c3d15fbf18a55d15369d8c548a319751fc41908377d418f984b230"}
Sep 12 02:40:46.228875 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.228843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6" event={"ID":"eee61980-927f-493a-8e93-fafe1447d816","Type":"ContainerStarted","Data":"6240a98cff704d4efc5dcd8fd32330aed1bda7496eec7fdae705d86686790235"}
Sep 12 02:40:46.230235 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.230207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6znn9" event={"ID":"4cb403fd-1ca8-4352-a7c7-ff62805c099c","Type":"ContainerStarted","Data":"1df086045af9bdcfa21dea13d44099c13402b12ac0204f1ea6faf5dc1040809c"}
Sep 12 02:40:46.231587 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.231561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvqz6" event={"ID":"a0e5173a-6ced-49c0-8c95-0fdaecf85e42","Type":"ContainerStarted","Data":"e0dd822699737ec20986f6365433dd62bed8e64532542f3cb3a287ddda20ff36"}
Sep 12 02:40:46.233391 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.233370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" event={"ID":"56373d50-df04-4236-80c5-9de7b5c6dc24","Type":"ContainerStarted","Data":"2db968f63a2fa8cff7758d557dc6d02ba8edb6c7f28c06919fe7c5b2ca03763d"}
Sep 12 02:40:46.233496 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.233411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" event={"ID":"56373d50-df04-4236-80c5-9de7b5c6dc24","Type":"ContainerStarted","Data":"fb435327703c002d62d6a45d4432611066f050210042421024528e55a369aa64"}
Sep 12 02:40:46.233496 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.233448    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:40:46.252933 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.252884    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" podStartSLOduration=263.252865036 podStartE2EDuration="4m23.252865036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:46.251365808 +0000 UTC m=+285.519399456" watchObservedRunningTime="2026-09-12 02:40:46.252865036 +0000 UTC m=+285.520898683"
Sep 12 02:40:46.372483 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.372403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"
Sep 12 02:40:46.667591 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:46.667431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr"]
Sep 12 02:40:46.933970 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:46.933936    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cc48e0_5a3b_4ad3_9d31_a3d750b9ccb4.slice/crio-0ddde480fc80b2400ffb89e15984340e7cbf185a41ba0ef27ccc049b6624b240 WatchSource:0}: Error finding container 0ddde480fc80b2400ffb89e15984340e7cbf185a41ba0ef27ccc049b6624b240: Status 404 returned error can't find the container with id 0ddde480fc80b2400ffb89e15984340e7cbf185a41ba0ef27ccc049b6624b240
Sep 12 02:40:47.239920 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:47.239838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr" event={"ID":"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4","Type":"ContainerStarted","Data":"0ddde480fc80b2400ffb89e15984340e7cbf185a41ba0ef27ccc049b6624b240"}
Sep 12 02:40:48.246421 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:48.246388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvqz6" event={"ID":"a0e5173a-6ced-49c0-8c95-0fdaecf85e42","Type":"ContainerStarted","Data":"13b79fd8037e1f0db71a14e501fdf863be5299dba6118b3046c8bdd9e5c4621b"}
Sep 12 02:40:49.251281 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.251244    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0e5173a-6ced-49c0-8c95-0fdaecf85e42" containerID="13b79fd8037e1f0db71a14e501fdf863be5299dba6118b3046c8bdd9e5c4621b" exitCode=0
Sep 12 02:40:49.251682 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.251329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvqz6" event={"ID":"a0e5173a-6ced-49c0-8c95-0fdaecf85e42","Type":"ContainerDied","Data":"13b79fd8037e1f0db71a14e501fdf863be5299dba6118b3046c8bdd9e5c4621b"}
Sep 12 02:40:49.253413 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.253392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr" event={"ID":"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4","Type":"ContainerStarted","Data":"4b49269ce6be3bdf122c91847a6345c18f9847e774f08f5a66d53c595f385df4"}
Sep 12 02:40:49.253508 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.253443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr" event={"ID":"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4","Type":"ContainerStarted","Data":"60938aad3566fab9c532e19fb4569466e8c30c0d69013b790017f1a5d6107b8f"}
Sep 12 02:40:49.253508 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.253472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr" event={"ID":"c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4","Type":"ContainerStarted","Data":"3a0172dac461f646432cca2dfcc852b3e67cdb47cda0b6d8a42887e060b28d8b"}
Sep 12 02:40:49.255606 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.255581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t" event={"ID":"269ee6aa-9c67-42a7-af13-617886fb4ad2","Type":"ContainerStarted","Data":"1235344b7b450c6bad6beedbfea1e9fb7416afce5276e1c9ec6c45e715c4bf94"}
Sep 12 02:40:49.259496 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.259470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6" event={"ID":"eee61980-927f-493a-8e93-fafe1447d816","Type":"ContainerStarted","Data":"a69df918b37e49ad5f9553fbc2ac13376dab956d6add3780338eb73131268767"}
Sep 12 02:40:49.261495 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.261464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6znn9" event={"ID":"4cb403fd-1ca8-4352-a7c7-ff62805c099c","Type":"ContainerStarted","Data":"0dd136984afdf31c2f90d10547891ee2bc62a28028814a458b261dfce55c14b8"}
Sep 12 02:40:49.261566 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.261510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6znn9" event={"ID":"4cb403fd-1ca8-4352-a7c7-ff62805c099c","Type":"ContainerStarted","Data":"c8b8ea7aaf854307d8e784fae511e49291c5f2e4e8b46b85710707f1260383dc"}
Sep 12 02:40:49.261754 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.261736    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6znn9"
Sep 12 02:40:49.285165 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.285083    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mwwbr" podStartSLOduration=2.761138795 podStartE2EDuration="4.28506727s" totalImagesPullingTime="1.523928475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:49.284033498 +0000 UTC m=+288.552067145" watchObservedRunningTime="2026-09-12 02:40:49.28506727 +0000 UTC m=+288.553100917"
Sep 12 02:40:49.304231 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.304192    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-4td4t" podStartSLOduration=2.396777673 podStartE2EDuration="4.304176964s" totalImagesPullingTime="1.907399291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:49.302944522 +0000 UTC m=+288.570978169" watchObservedRunningTime="2026-09-12 02:40:49.304176964 +0000 UTC m=+288.572210652"
Sep 12 02:40:49.317637 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.317589    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-tm7d6" podStartSLOduration=282.095789034 podStartE2EDuration="4m44.317574606s" totalImagesPullingTime="2.221785572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:49.316616379 +0000 UTC m=+288.584650028" watchObservedRunningTime="2026-09-12 02:40:49.317574606 +0000 UTC m=+288.585608253"
Sep 12 02:40:49.338219 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.338169    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6znn9" podStartSLOduration=251.757785149 podStartE2EDuration="4m14.338151164s" totalImagesPullingTime="2.580366015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:49.337624629 +0000 UTC m=+288.605658291" watchObservedRunningTime="2026-09-12 02:40:49.338151164 +0000 UTC m=+288.606184811"
Sep 12 02:40:49.771542 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.771505    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d4948d966-twqrk"]
Sep 12 02:40:49.774843 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.774826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:49.777427 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.777401    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 12 02:40:49.777427 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.777423    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-44phb05dpug2\""
Sep 12 02:40:49.778656 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.778632    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hfpvg\""
Sep 12 02:40:49.778749 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.778716    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 12 02:40:49.778842 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.778827    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 12 02:40:49.778953 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.778937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 12 02:40:49.784256 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.784227    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d4948d966-twqrk"]
Sep 12 02:40:49.943718 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.943677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/84111875-128a-40db-9be5-d15f57f18175-metrics-server-audit-profiles\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:49.943718 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.943720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-secret-metrics-server-tls\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:49.943978 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.943741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/84111875-128a-40db-9be5-d15f57f18175-audit-log\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:49.943978 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.943832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-secret-metrics-server-client-certs\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:49.943978 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.943871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcw8z\" (UniqueName: \"kubernetes.io/projected/84111875-128a-40db-9be5-d15f57f18175-kube-api-access-kcw8z\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:49.943978 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.943952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84111875-128a-40db-9be5-d15f57f18175-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:49.944164 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:49.944002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-client-ca-bundle\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.044993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-secret-metrics-server-tls\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/84111875-128a-40db-9be5-d15f57f18175-audit-log\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-secret-metrics-server-client-certs\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcw8z\" (UniqueName: \"kubernetes.io/projected/84111875-128a-40db-9be5-d15f57f18175-kube-api-access-kcw8z\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045425 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84111875-128a-40db-9be5-d15f57f18175-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045425 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-client-ca-bundle\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045425 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/84111875-128a-40db-9be5-d15f57f18175-metrics-server-audit-profiles\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.045573 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/84111875-128a-40db-9be5-d15f57f18175-audit-log\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.046120 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.045984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84111875-128a-40db-9be5-d15f57f18175-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.046357 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.046338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/84111875-128a-40db-9be5-d15f57f18175-metrics-server-audit-profiles\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.047890 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.047870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-secret-metrics-server-tls\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.047977 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.047868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-secret-metrics-server-client-certs\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.047977 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.047902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/84111875-128a-40db-9be5-d15f57f18175-client-ca-bundle\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.052523 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.052504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcw8z\" (UniqueName: \"kubernetes.io/projected/84111875-128a-40db-9be5-d15f57f18175-kube-api-access-kcw8z\") pod \"metrics-server-d4948d966-twqrk\" (UID: \"84111875-128a-40db-9be5-d15f57f18175\") " pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.084582 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.084535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:40:50.229176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.229145    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"]
Sep 12 02:40:50.233444 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.233426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:50.235588 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.235570    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 12 02:40:50.235703 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.235688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mnskf\""
Sep 12 02:40:50.240615 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.240590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"]
Sep 12 02:40:50.267755 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.267723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvqz6" event={"ID":"a0e5173a-6ced-49c0-8c95-0fdaecf85e42","Type":"ContainerStarted","Data":"85ec523f08cc48bd66b591c98308e92ce3cced748ed66e7d57c90d190c321363"}
Sep 12 02:40:50.268232 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.267775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvqz6" event={"ID":"a0e5173a-6ced-49c0-8c95-0fdaecf85e42","Type":"ContainerStarted","Data":"2cf5ab7e3c11fcfa130617af1a5920490da317a84387326c8720e9e2b67f3b09"}
Sep 12 02:40:50.284624 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.284579    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zvqz6" podStartSLOduration=2.977980046 podStartE2EDuration="5.284565419s" totalImagesPullingTime="2.228581837s" totalInitContainerRuntime="78.003536ms" isStatefulPod=true podCreationTimestamp="2026-09-12 02:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:50.28384784 +0000 UTC m=+289.551881486" watchObservedRunningTime="2026-09-12 02:40:50.284565419 +0000 UTC m=+289.552599066"
Sep 12 02:40:50.347541 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.347461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5tdvf\" (UID: \"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:50.347541 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.347518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d64\" (UniqueName: \"kubernetes.io/projected/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-kube-api-access-k6d64\") pod \"monitoring-plugin-5d85778b9d-5tdvf\" (UID: \"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:50.357397 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.357139    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d4948d966-twqrk"]
Sep 12 02:40:50.360270 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:50.360240    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84111875_128a_40db_9be5_d15f57f18175.slice/crio-4a26f363b93ae96721948bda8b6b437738851769f025036fb73e3ecc2c3e5e87 WatchSource:0}: Error finding container 4a26f363b93ae96721948bda8b6b437738851769f025036fb73e3ecc2c3e5e87: Status 404 returned error can't find the container with id 4a26f363b93ae96721948bda8b6b437738851769f025036fb73e3ecc2c3e5e87
Sep 12 02:40:50.448300 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.448264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d64\" (UniqueName: \"kubernetes.io/projected/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-kube-api-access-k6d64\") pod \"monitoring-plugin-5d85778b9d-5tdvf\" (UID: \"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:50.448475 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.448451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5tdvf\" (UID: \"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:50.448967 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:50.448940    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 12 02:40:50.449136 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:50.449118    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-monitoring-plugin-cert podName:204e049e-66fe-4b59-b5d1-bbdc9b3a35f3 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:50.949073901 +0000 UTC m=+290.217107526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-5tdvf" (UID: "204e049e-66fe-4b59-b5d1-bbdc9b3a35f3") : secret "monitoring-plugin-cert" not found
Sep 12 02:40:50.467026 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.466999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d64\" (UniqueName: \"kubernetes.io/projected/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-kube-api-access-k6d64\") pod \"monitoring-plugin-5d85778b9d-5tdvf\" (UID: \"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:50.953943 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.953848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5tdvf\" (UID: \"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:50.956417 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:50.956389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/204e049e-66fe-4b59-b5d1-bbdc9b3a35f3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5tdvf\" (UID: \"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:51.145391 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.145351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:51.273142 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.273042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d4948d966-twqrk" event={"ID":"84111875-128a-40db-9be5-d15f57f18175","Type":"ContainerStarted","Data":"4a26f363b93ae96721948bda8b6b437738851769f025036fb73e3ecc2c3e5e87"}
Sep 12 02:40:51.421132 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.420952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"]
Sep 12 02:40:51.707051 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.707019    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:40:51.710619 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.710600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.713219 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.713195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 12 02:40:51.714215 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:51.714192    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204e049e_66fe_4b59_b5d1_bbdc9b3a35f3.slice/crio-594fea079fa123436cd883aded184d4070b100c8f59c156cd05d1345780b8270 WatchSource:0}: Error finding container 594fea079fa123436cd883aded184d4070b100c8f59c156cd05d1345780b8270: Status 404 returned error can't find the container with id 594fea079fa123436cd883aded184d4070b100c8f59c156cd05d1345780b8270
Sep 12 02:40:51.714342 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714327    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-csi4fpgq2ibtv\""
Sep 12 02:40:51.714404 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714358    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 12 02:40:51.714500 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714482    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 12 02:40:51.714612 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714596    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4cxcl\""
Sep 12 02:40:51.714612 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714605    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 12 02:40:51.714700 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 12 02:40:51.714746 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714596    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 12 02:40:51.714746 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714714    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 12 02:40:51.715111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 12 02:40:51.715111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714941    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 12 02:40:51.715111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.714940    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 12 02:40:51.715371 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.715175    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 12 02:40:51.715371 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.715203    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 12 02:40:51.717044 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.717024    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 12 02:40:51.719532 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.719515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 12 02:40:51.729285 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.729264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:40:51.860253 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860373 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860373 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860373 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860540 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860540 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860540 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860653 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krqj7\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-kube-api-access-krqj7\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860653 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860653 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860748 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860748 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860748 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860748 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860748 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.860921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.860831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.961884 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.961784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.961884 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.961831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.961963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krqj7\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-kube-api-access-krqj7\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962504 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962504 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962504 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962504 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962504 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962728 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962728 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962728 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.962728 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.963204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.963204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.962589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.963204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.963045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.963204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.963077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.963204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.963138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.963204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.963170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.963719 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.963699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.965211 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.965182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.966829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.966803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.967371 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.967157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.967371 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.967227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.967371 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.967267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config-out\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.967371 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.967332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.967629 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.967575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.967886 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.967797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.968343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.968319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.969231 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.969203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.969490 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.969467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.969892 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.969872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.969963 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.969886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.970261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.970241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-web-config\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:51.971195 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:51.971172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krqj7\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-kube-api-access-krqj7\") pod \"prometheus-k8s-0\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:52.028926 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:52.028899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:40:52.279026 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:52.278914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d4948d966-twqrk" event={"ID":"84111875-128a-40db-9be5-d15f57f18175","Type":"ContainerStarted","Data":"6e2874bbb4e722bac9489e1dd72bda130313639a05c4a64d499fe72a50e42601"}
Sep 12 02:40:52.280636 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:52.280607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf" event={"ID":"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3","Type":"ContainerStarted","Data":"594fea079fa123436cd883aded184d4070b100c8f59c156cd05d1345780b8270"}
Sep 12 02:40:52.295774 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:52.295720    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d4948d966-twqrk" podStartSLOduration=1.89866833 podStartE2EDuration="3.295699295s" totalImagesPullingTime="1.397030965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:52.293998815 +0000 UTC m=+291.562032463" watchObservedRunningTime="2026-09-12 02:40:52.295699295 +0000 UTC m=+291.563732941"
Sep 12 02:40:52.309787 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:52.309727    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:40:52.312303 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:40:52.312274    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3325f0ef_70d4_465d_9fa2_1558cb06d75d.slice/crio-69ca9eee09bb4be447873d2592f85a46c9fb743af46abaf4ce7f3422ae4d41d3 WatchSource:0}: Error finding container 69ca9eee09bb4be447873d2592f85a46c9fb743af46abaf4ce7f3422ae4d41d3: Status 404 returned error can't find the container with id 69ca9eee09bb4be447873d2592f85a46c9fb743af46abaf4ce7f3422ae4d41d3
Sep 12 02:40:53.287232 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:53.287197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf" event={"ID":"204e049e-66fe-4b59-b5d1-bbdc9b3a35f3","Type":"ContainerStarted","Data":"2808299e017753f30e016618a65007aa4ac2b5c29fdce98b1e676b92dfda6c70"}
Sep 12 02:40:53.287695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:53.287615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:53.288819 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:53.288793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerStarted","Data":"69ca9eee09bb4be447873d2592f85a46c9fb743af46abaf4ce7f3422ae4d41d3"}
Sep 12 02:40:53.292731 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:53.292686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf"
Sep 12 02:40:53.302715 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:53.302676    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5tdvf" podStartSLOduration=2.174414439 podStartE2EDuration="3.302662089s" totalImagesPullingTime="1.12824765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:53.300767641 +0000 UTC m=+292.568801289" watchObservedRunningTime="2026-09-12 02:40:53.302662089 +0000 UTC m=+292.570695740"
Sep 12 02:40:54.294120 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:54.294061    2579 generic.go:356] "Generic (PLEG): container finished" podID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd" exitCode=0
Sep 12 02:40:54.294623 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:54.294148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"}
Sep 12 02:40:55.244307 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:40:55.244251    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-gncjn" podUID="c0eba23f-369a-4cec-954d-d152d68cc4bd"
Sep 12 02:40:58.316602 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:58.316556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerStarted","Data":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"}
Sep 12 02:40:58.317026 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:58.316648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerStarted","Data":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"}
Sep 12 02:40:59.270068 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:40:59.270035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6znn9"
Sep 12 02:41:00.329107 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:00.329052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerStarted","Data":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"}
Sep 12 02:41:00.329521 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:00.329177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerStarted","Data":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"}
Sep 12 02:41:00.329521 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:00.329204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerStarted","Data":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"}
Sep 12 02:41:00.329521 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:00.329230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerStarted","Data":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"}
Sep 12 02:41:00.362240 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:00.362186    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.354895486 podStartE2EDuration="9.362167444s" totalImagesPullingTime="5.989445894s" totalInitContainerRuntime="17.826064ms" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:41:00.359219637 +0000 UTC m=+299.627253294" watchObservedRunningTime="2026-09-12 02:41:00.362167444 +0000 UTC m=+299.630201093"
Sep 12 02:41:01.174204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:01.174177    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 12 02:41:02.029115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:02.029043    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:41:03.215725 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:03.215687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85d54dd6dd-xkg4p"]
Sep 12 02:41:03.220087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:03.220055    2579 patch_prober.go:28] interesting pod/image-registry-85d54dd6dd-xkg4p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 02:41:03.220255 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:03.220127    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:41:07.243036 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:07.242999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:41:10.085320 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:10.085266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:41:10.085736 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:10.085360    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:41:13.221081 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:13.221048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:41:23.428259 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:23.428166    2579 generic.go:356] "Generic (PLEG): container finished" podID="9fd11876-ba7c-415c-8b18-c450d94fa012" containerID="4459f5fb6adb6f50c2fa4bc1aed94f3c14d5228b6c39f5a7ea1b6bb342b4151b" exitCode=0
Sep 12 02:41:23.428671 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:23.428242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx" event={"ID":"9fd11876-ba7c-415c-8b18-c450d94fa012","Type":"ContainerDied","Data":"4459f5fb6adb6f50c2fa4bc1aed94f3c14d5228b6c39f5a7ea1b6bb342b4151b"}
Sep 12 02:41:23.428721 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:23.428678    2579 scope.go:117] "RemoveContainer" containerID="4459f5fb6adb6f50c2fa4bc1aed94f3c14d5228b6c39f5a7ea1b6bb342b4151b"
Sep 12 02:41:23.430118 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:23.430012    2579 generic.go:356] "Generic (PLEG): container finished" podID="4344e4db-5901-4556-916e-c38e0ca4f7a6" containerID="d3ced2e58324538080e9d2e1557488cd732494451cd5ac8d556efdd1e019643f" exitCode=0
Sep 12 02:41:23.430118 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:23.430056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-57tmp" event={"ID":"4344e4db-5901-4556-916e-c38e0ca4f7a6","Type":"ContainerDied","Data":"d3ced2e58324538080e9d2e1557488cd732494451cd5ac8d556efdd1e019643f"}
Sep 12 02:41:23.430415 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:23.430391    2579 scope.go:117] "RemoveContainer" containerID="d3ced2e58324538080e9d2e1557488cd732494451cd5ac8d556efdd1e019643f"
Sep 12 02:41:24.436482 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:24.436448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-57tmp" event={"ID":"4344e4db-5901-4556-916e-c38e0ca4f7a6","Type":"ContainerStarted","Data":"78d3fd936014b5185f5f80989b38fbc5b6a60856af1f768862c6d1d29c07ce09"}
Sep 12 02:41:24.438472 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:24.438446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-brgwx" event={"ID":"9fd11876-ba7c-415c-8b18-c450d94fa012","Type":"ContainerStarted","Data":"8a31e353fc492944be90348d7cf5dfb99b13113368a9eb4c7f7354695c4343de"}
Sep 12 02:41:28.236038 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:28.235978    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24" containerName="registry" containerID="cri-o://2db968f63a2fa8cff7758d557dc6d02ba8edb6c7f28c06919fe7c5b2ca03763d" gracePeriod=30
Sep 12 02:41:29.459133 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.459079    2579 generic.go:356] "Generic (PLEG): container finished" podID="56373d50-df04-4236-80c5-9de7b5c6dc24" containerID="2db968f63a2fa8cff7758d557dc6d02ba8edb6c7f28c06919fe7c5b2ca03763d" exitCode=0
Sep 12 02:41:29.459133 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.459136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" event={"ID":"56373d50-df04-4236-80c5-9de7b5c6dc24","Type":"ContainerDied","Data":"2db968f63a2fa8cff7758d557dc6d02ba8edb6c7f28c06919fe7c5b2ca03763d"}
Sep 12 02:41:29.495465 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.495427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:41:29.617227 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617124    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvlsf\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-kube-api-access-kvlsf\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617227 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-bound-sa-token\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617227 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-trusted-ca\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617243    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-installation-pull-secrets\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-image-registry-private-configuration\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617300    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56373d50-df04-4236-80c5-9de7b5c6dc24-ca-trust-extracted\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-certificates\") pod \"56373d50-df04-4236-80c5-9de7b5c6dc24\" (UID: \"56373d50-df04-4236-80c5-9de7b5c6dc24\") "
Sep 12 02:41:29.617934 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.617878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:41:29.618354 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.618313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:41:29.620753 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.620700    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:41:29.620881 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.620738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:41:29.620881 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.620782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-kube-api-access-kvlsf" (OuterVolumeSpecName: "kube-api-access-kvlsf") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "kube-api-access-kvlsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:41:29.620881 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.620839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:41:29.620999 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.620877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:41:29.626874 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.626837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56373d50-df04-4236-80c5-9de7b5c6dc24-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "56373d50-df04-4236-80c5-9de7b5c6dc24" (UID: "56373d50-df04-4236-80c5-9de7b5c6dc24"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:41:29.719050 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719005    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56373d50-df04-4236-80c5-9de7b5c6dc24-ca-trust-extracted\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:29.719050 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719039    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-certificates\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:29.719050 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719049    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvlsf\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-kube-api-access-kvlsf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:29.719050 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719058    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-bound-sa-token\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:29.719348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719067    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56373d50-df04-4236-80c5-9de7b5c6dc24-trusted-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:29.719348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719075    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-installation-pull-secrets\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:29.719348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719084    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56373d50-df04-4236-80c5-9de7b5c6dc24-image-registry-private-configuration\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:29.719348 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:29.719125    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56373d50-df04-4236-80c5-9de7b5c6dc24-registry-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:41:30.090621 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:30.090588    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:41:30.094693 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:30.094667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d4948d966-twqrk"
Sep 12 02:41:30.464868 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:30.464830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p"
Sep 12 02:41:30.465398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:30.464873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d54dd6dd-xkg4p" event={"ID":"56373d50-df04-4236-80c5-9de7b5c6dc24","Type":"ContainerDied","Data":"fb435327703c002d62d6a45d4432611066f050210042421024528e55a369aa64"}
Sep 12 02:41:30.465398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:30.464930    2579 scope.go:117] "RemoveContainer" containerID="2db968f63a2fa8cff7758d557dc6d02ba8edb6c7f28c06919fe7c5b2ca03763d"
Sep 12 02:41:30.486511 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:30.486479    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85d54dd6dd-xkg4p"]
Sep 12 02:41:30.487711 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:30.487688    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85d54dd6dd-xkg4p"]
Sep 12 02:41:31.248402 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:31.248368    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24" path="/var/lib/kubelet/pods/56373d50-df04-4236-80c5-9de7b5c6dc24/volumes"
Sep 12 02:41:38.497541 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:38.497504    2579 generic.go:356] "Generic (PLEG): container finished" podID="7fdbe5e6-27c1-4170-9ba4-e0ec798cb927" containerID="bc964729d1c76fda887de5ddabd8cacb9eed084feada335551d5816979504f2b" exitCode=0
Sep 12 02:41:38.497974 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:38.497579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww" event={"ID":"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927","Type":"ContainerDied","Data":"bc964729d1c76fda887de5ddabd8cacb9eed084feada335551d5816979504f2b"}
Sep 12 02:41:38.497974 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:38.497943    2579 scope.go:117] "RemoveContainer" containerID="bc964729d1c76fda887de5ddabd8cacb9eed084feada335551d5816979504f2b"
Sep 12 02:41:39.503216 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:39.503184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dnfww" event={"ID":"7fdbe5e6-27c1-4170-9ba4-e0ec798cb927","Type":"ContainerStarted","Data":"00a7f5d6dff4fcd8ebee3fe2b19349368a361af1e994605d1a48fc87e3110ca8"}
Sep 12 02:41:52.030047 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:52.030004    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:41:52.050022 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:52.049992    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:41:52.569742 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:41:52.569710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.055557 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.055519    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:42:10.056056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.055827    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="prometheus" containerID="cri-o://648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3" gracePeriod=600
Sep 12 02:42:10.056056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.055873    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9" gracePeriod=600
Sep 12 02:42:10.056056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.055919    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy" containerID="cri-o://9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d" gracePeriod=600
Sep 12 02:42:10.056056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.055912    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-web" containerID="cri-o://8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2" gracePeriod=600
Sep 12 02:42:10.056056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.055928    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="config-reloader" containerID="cri-o://dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936" gracePeriod=600
Sep 12 02:42:10.056056 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.055873    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="thanos-sidecar" containerID="cri-o://0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708" gracePeriod=600
Sep 12 02:42:10.299622 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.299596    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.452757 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krqj7\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-kube-api-access-krqj7\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.452948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-web-config\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.452948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-1\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.452948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452821    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-metrics-client-ca\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.452948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-0\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.452948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452872    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.452948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-2\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.452948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452939    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-db\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.452980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-kube-rbac-proxy\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-grpc-tls\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-serving-certs-ca-bundle\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-thanos-prometheus-http-client-file\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453141    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-kubelet-serving-ca-bundle\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-trusted-ca-bundle\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453345 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-tls-assets\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config-out\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-metrics-client-certs\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453423    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-tls\") pod \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\" (UID: \"3325f0ef-70d4-465d-9fa2-1558cb06d75d\") "
Sep 12 02:42:10.453837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:42:10.454028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453967    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.454028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.453986    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.454028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.454002    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-metrics-client-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.454907 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.454587    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:42:10.454907 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.454820    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:42:10.455458 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.455204    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:42:10.456041 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.456008    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-kube-api-access-krqj7" (OuterVolumeSpecName: "kube-api-access-krqj7") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "kube-api-access-krqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:42:10.456382 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.456348    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.456479 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.456460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:42:10.456632 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.456605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:42:10.456695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.456609    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.456769 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.456746    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:42:10.457361 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.457273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.457361 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.457297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.457620 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.457498    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.458771 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.458752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.458986 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.458973    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config" (OuterVolumeSpecName: "config") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.459185 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.459172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config-out" (OuterVolumeSpecName: "config-out") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:42:10.459235 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.459201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.467556 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.467529    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-web-config" (OuterVolumeSpecName: "web-config") pod "3325f0ef-70d4-465d-9fa2-1558cb06d75d" (UID: "3325f0ef-70d4-465d-9fa2-1558cb06d75d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:42:10.555312 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555279    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555312 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555308    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555312 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555318    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555328    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555337    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-tls-assets\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555345    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-config-out\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555352    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-metrics-client-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555361    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555369    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krqj7\" (UniqueName: \"kubernetes.io/projected/3325f0ef-70d4-465d-9fa2-1558cb06d75d-kube-api-access-krqj7\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555378    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-web-config\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555386    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555395    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555403    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3325f0ef-70d4-465d-9fa2-1558cb06d75d-prometheus-k8s-db\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555411    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-kube-rbac-proxy\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555419    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-secret-grpc-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555427    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3325f0ef-70d4-465d-9fa2-1558cb06d75d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.555515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.555435    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3325f0ef-70d4-465d-9fa2-1558cb06d75d-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:42:10.631695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631655    2579 generic.go:356] "Generic (PLEG): container finished" podID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9" exitCode=0
Sep 12 02:42:10.631695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631686    2579 generic.go:356] "Generic (PLEG): container finished" podID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d" exitCode=0
Sep 12 02:42:10.631695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631694    2579 generic.go:356] "Generic (PLEG): container finished" podID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2" exitCode=0
Sep 12 02:42:10.631695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631703    2579 generic.go:356] "Generic (PLEG): container finished" podID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708" exitCode=0
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631711    2579 generic.go:356] "Generic (PLEG): container finished" podID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936" exitCode=0
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631719    2579 generic.go:356] "Generic (PLEG): container finished" podID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3" exitCode=0
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631753    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"}
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"}
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"}
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631905    2579 scope.go:117] "RemoveContainer" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.631948 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"}
Sep 12 02:42:10.632258 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.631966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"}
Sep 12 02:42:10.632258 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.632007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"}
Sep 12 02:42:10.632258 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.632048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3325f0ef-70d4-465d-9fa2-1558cb06d75d","Type":"ContainerDied","Data":"69ca9eee09bb4be447873d2592f85a46c9fb743af46abaf4ce7f3422ae4d41d3"}
Sep 12 02:42:10.640288 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.640270    2579 scope.go:117] "RemoveContainer" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.647224 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.647207    2579 scope.go:117] "RemoveContainer" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.656520 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.656468    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:42:10.657456 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.657431    2579 scope.go:117] "RemoveContainer" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.662028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.662008    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:42:10.665313 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.665300    2579 scope.go:117] "RemoveContainer" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.672144 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.672126    2579 scope.go:117] "RemoveContainer" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.679419 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.679400    2579 scope.go:117] "RemoveContainer" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.683800 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.683775    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:42:10.684114 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684077    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="thanos-sidecar"
Sep 12 02:42:10.684192 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684115    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="thanos-sidecar"
Sep 12 02:42:10.684192 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684131    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy"
Sep 12 02:42:10.684192 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy"
Sep 12 02:42:10.684192 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684164    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="init-config-reloader"
Sep 12 02:42:10.684192 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684173    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="init-config-reloader"
Sep 12 02:42:10.684192 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684184    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24" containerName="registry"
Sep 12 02:42:10.684192 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684192    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24" containerName="registry"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684200    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="prometheus"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684208    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="prometheus"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684220    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="config-reloader"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684229    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="config-reloader"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684237    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-thanos"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684245    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-thanos"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684259    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-web"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684267    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-web"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684372    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684382    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-thanos"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684393    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="56373d50-df04-4236-80c5-9de7b5c6dc24" containerName="registry"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684402    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="kube-rbac-proxy-web"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684413    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="thanos-sidecar"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684422    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="prometheus"
Sep 12 02:42:10.684513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.684430    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" containerName="config-reloader"
Sep 12 02:42:10.686482 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.686462    2579 scope.go:117] "RemoveContainer" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.686786 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:42:10.686766    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": container with ID starting with b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9 not found: ID does not exist" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.686839 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.686801    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"} err="failed to get container status \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": rpc error: code = NotFound desc = could not find container \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": container with ID starting with b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9 not found: ID does not exist"
Sep 12 02:42:10.686882 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.686839    2579 scope.go:117] "RemoveContainer" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.687084 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:42:10.687066    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": container with ID starting with 9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d not found: ID does not exist" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.687275 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687125    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"} err="failed to get container status \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": rpc error: code = NotFound desc = could not find container \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": container with ID starting with 9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d not found: ID does not exist"
Sep 12 02:42:10.687275 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687151    2579 scope.go:117] "RemoveContainer" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.687368 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:42:10.687351    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": container with ID starting with 8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2 not found: ID does not exist" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.687420 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687374    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"} err="failed to get container status \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": rpc error: code = NotFound desc = could not find container \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": container with ID starting with 8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2 not found: ID does not exist"
Sep 12 02:42:10.687420 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687389    2579 scope.go:117] "RemoveContainer" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.687644 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:42:10.687627    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": container with ID starting with 0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708 not found: ID does not exist" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.687690 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687649    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"} err="failed to get container status \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": rpc error: code = NotFound desc = could not find container \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": container with ID starting with 0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708 not found: ID does not exist"
Sep 12 02:42:10.687690 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687662    2579 scope.go:117] "RemoveContainer" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.687913 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:42:10.687897    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": container with ID starting with dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936 not found: ID does not exist" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.687979 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687922    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"} err="failed to get container status \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": rpc error: code = NotFound desc = could not find container \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": container with ID starting with dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936 not found: ID does not exist"
Sep 12 02:42:10.687979 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.687944    2579 scope.go:117] "RemoveContainer" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.688258 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:42:10.688238    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": container with ID starting with 648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3 not found: ID does not exist" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.688296 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688264    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"} err="failed to get container status \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": rpc error: code = NotFound desc = could not find container \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": container with ID starting with 648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3 not found: ID does not exist"
Sep 12 02:42:10.688296 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688282    2579 scope.go:117] "RemoveContainer" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.688513 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:42:10.688497    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": container with ID starting with 7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd not found: ID does not exist" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.688551 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688519    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"} err="failed to get container status \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": rpc error: code = NotFound desc = could not find container \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": container with ID starting with 7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd not found: ID does not exist"
Sep 12 02:42:10.688551 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688533    2579 scope.go:117] "RemoveContainer" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.688729 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688714    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"} err="failed to get container status \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": rpc error: code = NotFound desc = could not find container \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": container with ID starting with b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9 not found: ID does not exist"
Sep 12 02:42:10.688774 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688730    2579 scope.go:117] "RemoveContainer" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.688917 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688901    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"} err="failed to get container status \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": rpc error: code = NotFound desc = could not find container \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": container with ID starting with 9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d not found: ID does not exist"
Sep 12 02:42:10.688957 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.688918    2579 scope.go:117] "RemoveContainer" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.689156 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689141    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"} err="failed to get container status \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": rpc error: code = NotFound desc = could not find container \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": container with ID starting with 8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2 not found: ID does not exist"
Sep 12 02:42:10.689156 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689155    2579 scope.go:117] "RemoveContainer" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.689337 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689322    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"} err="failed to get container status \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": rpc error: code = NotFound desc = could not find container \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": container with ID starting with 0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708 not found: ID does not exist"
Sep 12 02:42:10.689381 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689338    2579 scope.go:117] "RemoveContainer" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.689557 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689541    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"} err="failed to get container status \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": rpc error: code = NotFound desc = could not find container \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": container with ID starting with dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936 not found: ID does not exist"
Sep 12 02:42:10.689557 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689557    2579 scope.go:117] "RemoveContainer" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.689792 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689773    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"} err="failed to get container status \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": rpc error: code = NotFound desc = could not find container \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": container with ID starting with 648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3 not found: ID does not exist"
Sep 12 02:42:10.689863 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689794    2579 scope.go:117] "RemoveContainer" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.689863 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.689847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.690205 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.690013    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"} err="failed to get container status \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": rpc error: code = NotFound desc = could not find container \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": container with ID starting with 7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd not found: ID does not exist"
Sep 12 02:42:10.690205 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.690034    2579 scope.go:117] "RemoveContainer" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.690519 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.690497    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"} err="failed to get container status \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": rpc error: code = NotFound desc = could not find container \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": container with ID starting with b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9 not found: ID does not exist"
Sep 12 02:42:10.690590 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.690519    2579 scope.go:117] "RemoveContainer" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.690782 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.690764    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"} err="failed to get container status \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": rpc error: code = NotFound desc = could not find container \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": container with ID starting with 9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d not found: ID does not exist"
Sep 12 02:42:10.690858 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.690784    2579 scope.go:117] "RemoveContainer" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.691017 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.690995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"} err="failed to get container status \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": rpc error: code = NotFound desc = could not find container \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": container with ID starting with 8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2 not found: ID does not exist"
Sep 12 02:42:10.691066 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.691020    2579 scope.go:117] "RemoveContainer" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.691294 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.691276    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"} err="failed to get container status \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": rpc error: code = NotFound desc = could not find container \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": container with ID starting with 0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708 not found: ID does not exist"
Sep 12 02:42:10.691376 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.691295    2579 scope.go:117] "RemoveContainer" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.691576 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.691559    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"} err="failed to get container status \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": rpc error: code = NotFound desc = could not find container \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": container with ID starting with dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936 not found: ID does not exist"
Sep 12 02:42:10.691625 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.691577    2579 scope.go:117] "RemoveContainer" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.691814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.691792    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"} err="failed to get container status \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": rpc error: code = NotFound desc = could not find container \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": container with ID starting with 648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3 not found: ID does not exist"
Sep 12 02:42:10.691881 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.691815    2579 scope.go:117] "RemoveContainer" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.692146 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692128    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"} err="failed to get container status \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": rpc error: code = NotFound desc = could not find container \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": container with ID starting with 7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd not found: ID does not exist"
Sep 12 02:42:10.692213 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692145    2579 scope.go:117] "RemoveContainer" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.692386 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692369    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"} err="failed to get container status \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": rpc error: code = NotFound desc = could not find container \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": container with ID starting with b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9 not found: ID does not exist"
Sep 12 02:42:10.692460 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692388    2579 scope.go:117] "RemoveContainer" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.692692 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692653    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"} err="failed to get container status \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": rpc error: code = NotFound desc = could not find container \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": container with ID starting with 9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d not found: ID does not exist"
Sep 12 02:42:10.692789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692695    2579 scope.go:117] "RemoveContainer" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.692789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692705    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 12 02:42:10.692789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692709    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 12 02:42:10.692789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692772    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 12 02:42:10.693012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 12 02:42:10.693012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 12 02:42:10.693012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.692890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 12 02:42:10.693238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693032    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 12 02:42:10.693238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693154    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 12 02:42:10.693238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693169    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"} err="failed to get container status \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": rpc error: code = NotFound desc = could not find container \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": container with ID starting with 8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2 not found: ID does not exist"
Sep 12 02:42:10.693238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693189    2579 scope.go:117] "RemoveContainer" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.693238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 12 02:42:10.693530 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693298    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-csi4fpgq2ibtv\""
Sep 12 02:42:10.693530 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693309    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 12 02:42:10.693661 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693550    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"} err="failed to get container status \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": rpc error: code = NotFound desc = could not find container \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": container with ID starting with 0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708 not found: ID does not exist"
Sep 12 02:42:10.693661 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693573    2579 scope.go:117] "RemoveContainer" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.693661 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 12 02:42:10.693661 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693636    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 12 02:42:10.693881 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693639    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4cxcl\""
Sep 12 02:42:10.693940 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693900    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"} err="failed to get container status \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": rpc error: code = NotFound desc = could not find container \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": container with ID starting with dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936 not found: ID does not exist"
Sep 12 02:42:10.693940 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.693916    2579 scope.go:117] "RemoveContainer" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.694196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.694178    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"} err="failed to get container status \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": rpc error: code = NotFound desc = could not find container \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": container with ID starting with 648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3 not found: ID does not exist"
Sep 12 02:42:10.694256 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.694198    2579 scope.go:117] "RemoveContainer" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.694872    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"} err="failed to get container status \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": rpc error: code = NotFound desc = could not find container \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": container with ID starting with 7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.694903    2579 scope.go:117] "RemoveContainer" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.697404    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"} err="failed to get container status \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": rpc error: code = NotFound desc = could not find container \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": container with ID starting with b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9 not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.697433    2579 scope.go:117] "RemoveContainer" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.697703    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"} err="failed to get container status \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": rpc error: code = NotFound desc = could not find container \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": container with ID starting with 9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.697724    2579 scope.go:117] "RemoveContainer" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.697999    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"} err="failed to get container status \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": rpc error: code = NotFound desc = could not find container \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": container with ID starting with 8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2 not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.698019    2579 scope.go:117] "RemoveContainer" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.698278    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"} err="failed to get container status \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": rpc error: code = NotFound desc = could not find container \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": container with ID starting with 0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708 not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.698304    2579 scope.go:117] "RemoveContainer" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.698553    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"} err="failed to get container status \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": rpc error: code = NotFound desc = could not find container \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": container with ID starting with dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936 not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.698573    2579 scope.go:117] "RemoveContainer" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.698819    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"} err="failed to get container status \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": rpc error: code = NotFound desc = could not find container \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": container with ID starting with 648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3 not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.698840    2579 scope.go:117] "RemoveContainer" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.699116    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"} err="failed to get container status \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": rpc error: code = NotFound desc = could not find container \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": container with ID starting with 7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd not found: ID does not exist"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.699138    2579 scope.go:117] "RemoveContainer" containerID="b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"
Sep 12 02:42:10.700111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.699375    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9"} err="failed to get container status \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": rpc error: code = NotFound desc = could not find container \"b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9\": container with ID starting with b8c3f7863fb21f08d237db06a678d6275f11e06979768f3c2e04d4180917c7a9 not found: ID does not exist"
Sep 12 02:42:10.701330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.699471    2579 scope.go:117] "RemoveContainer" containerID="9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"
Sep 12 02:42:10.701330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.699714    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d"} err="failed to get container status \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": rpc error: code = NotFound desc = could not find container \"9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d\": container with ID starting with 9733883e05cb0c8c59f1164512d4dccb8a15f3e66e3ac15ebdc03952a9487f7d not found: ID does not exist"
Sep 12 02:42:10.701330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.699770    2579 scope.go:117] "RemoveContainer" containerID="8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"
Sep 12 02:42:10.701330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.700074    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2"} err="failed to get container status \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": rpc error: code = NotFound desc = could not find container \"8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2\": container with ID starting with 8d09e92edc108ffc8e0cac91353ddfc1d047d0bf485853010324255a3615a5f2 not found: ID does not exist"
Sep 12 02:42:10.701330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.700123    2579 scope.go:117] "RemoveContainer" containerID="0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"
Sep 12 02:42:10.701330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.700405    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708"} err="failed to get container status \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": rpc error: code = NotFound desc = could not find container \"0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708\": container with ID starting with 0c377c69d0005ae883a02fad8da93b048e31f5ad8757dde49d51fe7fe2351708 not found: ID does not exist"
Sep 12 02:42:10.701330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.700431    2579 scope.go:117] "RemoveContainer" containerID="dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"
Sep 12 02:42:10.702736 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.702007    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936"} err="failed to get container status \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": rpc error: code = NotFound desc = could not find container \"dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936\": container with ID starting with dc218825affebde48be6c091668abcda3222cfa924a620287fc6240750d20936 not found: ID does not exist"
Sep 12 02:42:10.702736 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.702032    2579 scope.go:117] "RemoveContainer" containerID="648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"
Sep 12 02:42:10.702736 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.702224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 12 02:42:10.702895 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.702814    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3"} err="failed to get container status \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": rpc error: code = NotFound desc = could not find container \"648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3\": container with ID starting with 648d1b8e2050fa3d1c0e8fca871b261f933eff34ecdc96991a5dadf3e88c15f3 not found: ID does not exist"
Sep 12 02:42:10.702895 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.702834    2579 scope.go:117] "RemoveContainer" containerID="7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"
Sep 12 02:42:10.703598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.703566    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd"} err="failed to get container status \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": rpc error: code = NotFound desc = could not find container \"7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd\": container with ID starting with 7b881335bc3ec7a6960375fb93296df498215d9931c673853e13e3af74fe97fd not found: ID does not exist"
Sep 12 02:42:10.705028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.705006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:42:10.705708 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.705691    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 12 02:42:10.757787 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.757787 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.757972 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ce6e596-cf48-42e5-bd2b-39c90e973d61-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.757972 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.757972 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.757972 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.757972 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758141 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.757994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-config\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758141 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758141 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758141 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758141 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqww9\" (UniqueName: \"kubernetes.io/projected/5ce6e596-cf48-42e5-bd2b-39c90e973d61-kube-api-access-sqww9\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758141 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758311 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758311 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758311 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758311 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758311 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ce6e596-cf48-42e5-bd2b-39c90e973d61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758311 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.758526 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.758315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859533 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ce6e596-cf48-42e5-bd2b-39c90e973d61-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859533 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859700 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859700 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859700 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859700 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-config\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859700 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.859916 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.859717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.860428 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.860684 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqww9\" (UniqueName: \"kubernetes.io/projected/5ce6e596-cf48-42e5-bd2b-39c90e973d61-kube-api-access-sqww9\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.860980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.861015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.861044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ce6e596-cf48-42e5-bd2b-39c90e973d61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.861069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.861118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.861176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.861200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.861837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.861518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.862695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.862590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ce6e596-cf48-42e5-bd2b-39c90e973d61-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.863368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.863048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.863368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.863078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.863368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.863178    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.863368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.863273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-config\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.863368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.863336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.863883 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.863852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.863960 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.863911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.864204 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.864175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.864687 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.864658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6e596-cf48-42e5-bd2b-39c90e973d61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.864819 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.864799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ce6e596-cf48-42e5-bd2b-39c90e973d61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.865502 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.865479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.865890 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.865865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.866201 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.866185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.866251 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.866240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ce6e596-cf48-42e5-bd2b-39c90e973d61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:10.868858 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:10.868841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqww9\" (UniqueName: \"kubernetes.io/projected/5ce6e596-cf48-42e5-bd2b-39c90e973d61-kube-api-access-sqww9\") pod \"prometheus-k8s-0\" (UID: \"5ce6e596-cf48-42e5-bd2b-39c90e973d61\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:11.002933 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:11.002847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:11.248457 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:11.248425    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3325f0ef-70d4-465d-9fa2-1558cb06d75d" path="/var/lib/kubelet/pods/3325f0ef-70d4-465d-9fa2-1558cb06d75d/volumes"
Sep 12 02:42:11.288997 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:11.288914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 12 02:42:11.291735 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:42:11.291709    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce6e596_cf48_42e5_bd2b_39c90e973d61.slice/crio-d79d26b8e6d2a990dd2317ce668f8dde22216653f4ee66b65a1b09d5d214b04d WatchSource:0}: Error finding container d79d26b8e6d2a990dd2317ce668f8dde22216653f4ee66b65a1b09d5d214b04d: Status 404 returned error can't find the container with id d79d26b8e6d2a990dd2317ce668f8dde22216653f4ee66b65a1b09d5d214b04d
Sep 12 02:42:11.639254 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:11.639158    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ce6e596-cf48-42e5-bd2b-39c90e973d61" containerID="b5f5db24f150ecb5b541f261a80fe92ee870e378dd2ce25672b72783a11bbdf8" exitCode=0
Sep 12 02:42:11.639254 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:11.639245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerDied","Data":"b5f5db24f150ecb5b541f261a80fe92ee870e378dd2ce25672b72783a11bbdf8"}
Sep 12 02:42:11.639475 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:11.639322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerStarted","Data":"d79d26b8e6d2a990dd2317ce668f8dde22216653f4ee66b65a1b09d5d214b04d"}
Sep 12 02:42:12.647700 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:12.647655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerStarted","Data":"8186fe0ea66b5ca5fa451f64cb3f0fefe1911449fe7406d1ac09e7dc0b549a90"}
Sep 12 02:42:12.648119 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:12.647732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerStarted","Data":"6e22514741e1fb946cbe05ae0fa2a403319bf4a7551e581eaf2339592e299d24"}
Sep 12 02:42:12.648119 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:12.647768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerStarted","Data":"2a76b34201a5d43046d3e605a55343e6b3ed188b9251ba632bc1263933c140aa"}
Sep 12 02:42:12.648119 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:12.647796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerStarted","Data":"da7dd00282922fdec53164d774b0c6f69404ca7345bd21bc738f1776ffbbdd01"}
Sep 12 02:42:12.648119 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:12.647821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerStarted","Data":"5fc0bc00a8455314e2c6062349fe699d93c7db8e606cd05b03004b684820d2fe"}
Sep 12 02:42:12.648119 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:12.647864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ce6e596-cf48-42e5-bd2b-39c90e973d61","Type":"ContainerStarted","Data":"5922a3f8cfc799a4e443fd35b623b6ce414667c23034404718e26dfc47cd264d"}
Sep 12 02:42:12.684322 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:12.684239    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.66366052 podStartE2EDuration="2.684213929s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.553409ms" isStatefulPod=false podCreationTimestamp="2026-09-12 02:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:42:12.681160751 +0000 UTC m=+371.949194398" watchObservedRunningTime="2026-09-12 02:42:12.684213929 +0000 UTC m=+371.952247580"
Sep 12 02:42:15.305757 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:15.305698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:42:15.308266 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:15.308236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0eba23f-369a-4cec-954d-d152d68cc4bd-metrics-certs\") pod \"network-metrics-daemon-gncjn\" (UID: \"c0eba23f-369a-4cec-954d-d152d68cc4bd\") " pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:42:15.346861 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:15.346832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9jkq\""
Sep 12 02:42:15.354880 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:15.354856    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gncjn"
Sep 12 02:42:15.626333 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:15.626124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gncjn"]
Sep 12 02:42:15.629118 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:42:15.629068    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eba23f_369a_4cec_954d_d152d68cc4bd.slice/crio-f5fb3b0e270ef01b4d34171218ea0a3b5f70624212e1603b212004512742c276 WatchSource:0}: Error finding container f5fb3b0e270ef01b4d34171218ea0a3b5f70624212e1603b212004512742c276: Status 404 returned error can't find the container with id f5fb3b0e270ef01b4d34171218ea0a3b5f70624212e1603b212004512742c276
Sep 12 02:42:15.631263 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:15.631243    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:42:15.661620 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:15.661593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gncjn" event={"ID":"c0eba23f-369a-4cec-954d-d152d68cc4bd","Type":"ContainerStarted","Data":"f5fb3b0e270ef01b4d34171218ea0a3b5f70624212e1603b212004512742c276"}
Sep 12 02:42:16.003737 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:16.003701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:42:17.673925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:17.673883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gncjn" event={"ID":"c0eba23f-369a-4cec-954d-d152d68cc4bd","Type":"ContainerStarted","Data":"9cc4fab9d0d6c457e39aab81a65c8897d39ceda23ea046c403ff914eb6a00298"}
Sep 12 02:42:17.674455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:17.673948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gncjn" event={"ID":"c0eba23f-369a-4cec-954d-d152d68cc4bd","Type":"ContainerStarted","Data":"560409328d9683bb9d0bf2b96cd9f9391188865edc8060cebcf43b392f82af82"}
Sep 12 02:42:17.690139 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:42:17.690053    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gncjn" podStartSLOduration=375.591431064 podStartE2EDuration="6m16.690034349s" totalImagesPullingTime="1.098603285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:42:17.690003932 +0000 UTC m=+376.958037580" watchObservedRunningTime="2026-09-12 02:42:17.690034349 +0000 UTC m=+376.958067998"
Sep 12 02:43:11.003919 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:43:11.003875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:43:11.020917 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:43:11.020888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:43:11.910151 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:43:11.910119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 12 02:45:07.851625 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.851549    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"]
Sep 12 02:45:07.854917 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.854897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:07.857408 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.857383    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 12 02:45:07.858337 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.858323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-lt5lr\""
Sep 12 02:45:07.858410 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.858329    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:45:07.864407 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.864385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"]
Sep 12 02:45:07.991351 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.991318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49bfe09-3647-42ec-a63c-35c0b592972c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vv8wg\" (UID: \"d49bfe09-3647-42ec-a63c-35c0b592972c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:07.991529 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:07.991373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcwq\" (UniqueName: \"kubernetes.io/projected/d49bfe09-3647-42ec-a63c-35c0b592972c-kube-api-access-jwcwq\") pod \"openshift-lws-operator-fd8ccff4c-vv8wg\" (UID: \"d49bfe09-3647-42ec-a63c-35c0b592972c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:08.092044 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:08.092007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcwq\" (UniqueName: \"kubernetes.io/projected/d49bfe09-3647-42ec-a63c-35c0b592972c-kube-api-access-jwcwq\") pod \"openshift-lws-operator-fd8ccff4c-vv8wg\" (UID: \"d49bfe09-3647-42ec-a63c-35c0b592972c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:08.092256 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:08.092129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49bfe09-3647-42ec-a63c-35c0b592972c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vv8wg\" (UID: \"d49bfe09-3647-42ec-a63c-35c0b592972c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:08.092507 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:08.092487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d49bfe09-3647-42ec-a63c-35c0b592972c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vv8wg\" (UID: \"d49bfe09-3647-42ec-a63c-35c0b592972c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:08.100592 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:08.100558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcwq\" (UniqueName: \"kubernetes.io/projected/d49bfe09-3647-42ec-a63c-35c0b592972c-kube-api-access-jwcwq\") pod \"openshift-lws-operator-fd8ccff4c-vv8wg\" (UID: \"d49bfe09-3647-42ec-a63c-35c0b592972c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:08.181540 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:08.181512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"
Sep 12 02:45:08.453240 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:08.453157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg"]
Sep 12 02:45:08.456972 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:45:08.456942    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd49bfe09_3647_42ec_a63c_35c0b592972c.slice/crio-c639b1a6841a2a219ed5ac3365699732b56edd2d21247a3b39622d8f255e096c WatchSource:0}: Error finding container c639b1a6841a2a219ed5ac3365699732b56edd2d21247a3b39622d8f255e096c: Status 404 returned error can't find the container with id c639b1a6841a2a219ed5ac3365699732b56edd2d21247a3b39622d8f255e096c
Sep 12 02:45:09.365158 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:09.365116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg" event={"ID":"d49bfe09-3647-42ec-a63c-35c0b592972c","Type":"ContainerStarted","Data":"c639b1a6841a2a219ed5ac3365699732b56edd2d21247a3b39622d8f255e096c"}
Sep 12 02:45:11.375051 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:11.375008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg" event={"ID":"d49bfe09-3647-42ec-a63c-35c0b592972c","Type":"ContainerStarted","Data":"d807c779acc9dac3940a3aba1d9bcc90a0ca4fe8fd580f926d49ab0b0e466b2f"}
Sep 12 02:45:11.392575 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:11.392526    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vv8wg" podStartSLOduration=1.996123695 podStartE2EDuration="4.392509283s" totalImagesPullingTime="2.396385588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:11.389911571 +0000 UTC m=+550.657945230" watchObservedRunningTime="2026-09-12 02:45:11.392509283 +0000 UTC m=+550.660542930"
Sep 12 02:45:28.110319 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.110282    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"]
Sep 12 02:45:28.113558 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.113538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.116681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.116631    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 12 02:45:28.120012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.119989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 12 02:45:28.120782 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.120765    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vpxgb\""
Sep 12 02:45:28.120879 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.120780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 12 02:45:28.132130 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.132104    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"]
Sep 12 02:45:28.155678 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.155643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-metrics-cert\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.155812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.155698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf59v\" (UniqueName: \"kubernetes.io/projected/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-kube-api-access-hf59v\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.155854 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.155816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-manager-config\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.155854 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.155845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-cert\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.256634 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.256603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-manager-config\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.256797 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.256643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-cert\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.256797 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.256774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-metrics-cert\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.256886 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.256837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf59v\" (UniqueName: \"kubernetes.io/projected/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-kube-api-access-hf59v\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.257485 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.257456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-manager-config\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.259393 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.259366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-cert\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.259510 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.259406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-metrics-cert\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.270028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.270003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf59v\" (UniqueName: \"kubernetes.io/projected/925b343b-f03a-4e31-b4f8-df7cdf1a4e1b-kube-api-access-hf59v\") pod \"lws-controller-manager-6645c7894d-xt89l\" (UID: \"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b\") " pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.423643 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.423534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:28.701901 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:28.701866    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"]
Sep 12 02:45:28.705654 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:45:28.705631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925b343b_f03a_4e31_b4f8_df7cdf1a4e1b.slice/crio-f3ac9f49046ad2e87f308ca567a8fa5d0cf4419cc6ba673a3bbef9a486a7dc25 WatchSource:0}: Error finding container f3ac9f49046ad2e87f308ca567a8fa5d0cf4419cc6ba673a3bbef9a486a7dc25: Status 404 returned error can't find the container with id f3ac9f49046ad2e87f308ca567a8fa5d0cf4419cc6ba673a3bbef9a486a7dc25
Sep 12 02:45:29.224467 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.224436    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"]
Sep 12 02:45:29.228989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.228968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.231443 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.231423    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 12 02:45:29.231576 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.231561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dx684\""
Sep 12 02:45:29.231827 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.231811    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 12 02:45:29.231874 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.231815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 12 02:45:29.232141 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.232127    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 12 02:45:29.250319 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.250295    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"]
Sep 12 02:45:29.265706 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.265679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.270352 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.265741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvn5q\" (UniqueName: \"kubernetes.io/projected/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-kube-api-access-jvn5q\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.270352 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.266318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-tmp\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.270352 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.266449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-manifests\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.270352 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.266515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-webhook-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.367065 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.367022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.367266 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.367079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvn5q\" (UniqueName: \"kubernetes.io/projected/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-kube-api-access-jvn5q\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.367315 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.367279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-tmp\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.367366 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.367350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-manifests\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.367403 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.367379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-webhook-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.367795 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.367771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-manifests\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.369693 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.369665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.369693 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.369681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-tmp\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.369897 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.369882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-webhook-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.375961 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.375931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvn5q\" (UniqueName: \"kubernetes.io/projected/b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0-kube-api-access-jvn5q\") pod \"opendatahub-operator-controller-manager-774995dcdd-6rhcs\" (UID: \"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.451133 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.451081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l" event={"ID":"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b","Type":"ContainerStarted","Data":"f3ac9f49046ad2e87f308ca567a8fa5d0cf4419cc6ba673a3bbef9a486a7dc25"}
Sep 12 02:45:29.539956 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.539864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:29.821313 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:29.821231    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"]
Sep 12 02:45:29.825978 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:45:29.825947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c5ff9b_074d_4c8f_a0f5_b2d52b4e7ea0.slice/crio-ddaa5e2bec3f3a269b5b09e5661209179f913186de3ba98a5ac686ea81ffcb99 WatchSource:0}: Error finding container ddaa5e2bec3f3a269b5b09e5661209179f913186de3ba98a5ac686ea81ffcb99: Status 404 returned error can't find the container with id ddaa5e2bec3f3a269b5b09e5661209179f913186de3ba98a5ac686ea81ffcb99
Sep 12 02:45:30.456893 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:30.456859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs" event={"ID":"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0","Type":"ContainerStarted","Data":"ddaa5e2bec3f3a269b5b09e5661209179f913186de3ba98a5ac686ea81ffcb99"}
Sep 12 02:45:31.465928 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:31.465584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l" event={"ID":"925b343b-f03a-4e31-b4f8-df7cdf1a4e1b","Type":"ContainerStarted","Data":"6cfddeee5a9c15dc6edfe0599d64e2fef43ae4236464d54249c230f0aadc23d3"}
Sep 12 02:45:31.465928 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:31.465739    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:31.481768 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:31.481708    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l" podStartSLOduration=1.332679963 podStartE2EDuration="3.481686553s" totalImagesPullingTime="2.14900659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:31.481079702 +0000 UTC m=+570.749113350" watchObservedRunningTime="2026-09-12 02:45:31.481686553 +0000 UTC m=+570.749720212"
Sep 12 02:45:33.476827 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:33.476788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs" event={"ID":"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0","Type":"ContainerStarted","Data":"a134c7be987a52044cb373ebc684a6c56cb50ac22c0d333b261346fc91c1a08c"}
Sep 12 02:45:34.482787 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:34.482743    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0" containerID="a134c7be987a52044cb373ebc684a6c56cb50ac22c0d333b261346fc91c1a08c" exitCode=0
Sep 12 02:45:34.483153 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:34.482822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs" event={"ID":"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0","Type":"ContainerDied","Data":"a134c7be987a52044cb373ebc684a6c56cb50ac22c0d333b261346fc91c1a08c"}
Sep 12 02:45:35.489683 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:35.489650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs" event={"ID":"b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0","Type":"ContainerStarted","Data":"83f53f9e7b4c1896993dcfb064795cc7283227e38695d57263f639905754226c"}
Sep 12 02:45:35.490070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:35.489750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:35.509497 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:35.509445    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs" podStartSLOduration=1.652737288 podStartE2EDuration="6.509428627s" totalImagesPullingTime="3.820397204s" totalInitContainerRuntime="1.036294135s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:35.508344062 +0000 UTC m=+574.776377709" watchObservedRunningTime="2026-09-12 02:45:35.509428627 +0000 UTC m=+574.777462273"
Sep 12 02:45:42.473573 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:42.473540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6645c7894d-xt89l"
Sep 12 02:45:46.497078 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:46.497047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-6rhcs"
Sep 12 02:45:50.994216 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:50.994138    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"]
Sep 12 02:45:51.021955 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.021922    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"]
Sep 12 02:45:51.022190 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.022079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.024685 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.024652    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 12 02:45:51.024824 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.024700    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 02:45:51.025757 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.025739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sll9c\""
Sep 12 02:45:51.025991 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.025817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 02:45:51.026129 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.025817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 12 02:45:51.162661 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.162618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f0e59c-c794-432d-a95d-de3483af1be3-tls-certs\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.162847 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.162731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83f0e59c-c794-432d-a95d-de3483af1be3-tmp\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.162847 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.162759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm87n\" (UniqueName: \"kubernetes.io/projected/83f0e59c-c794-432d-a95d-de3483af1be3-kube-api-access-dm87n\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.264130 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.264031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83f0e59c-c794-432d-a95d-de3483af1be3-tmp\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.264130 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.264067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm87n\" (UniqueName: \"kubernetes.io/projected/83f0e59c-c794-432d-a95d-de3483af1be3-kube-api-access-dm87n\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.264314 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.264138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f0e59c-c794-432d-a95d-de3483af1be3-tls-certs\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.266600 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.266563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83f0e59c-c794-432d-a95d-de3483af1be3-tmp\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.266786 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.266768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f0e59c-c794-432d-a95d-de3483af1be3-tls-certs\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.272598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.272572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm87n\" (UniqueName: \"kubernetes.io/projected/83f0e59c-c794-432d-a95d-de3483af1be3-kube-api-access-dm87n\") pod \"kube-auth-proxy-84dbf54759-gw4sv\" (UID: \"83f0e59c-c794-432d-a95d-de3483af1be3\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.333307 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.333278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"
Sep 12 02:45:51.613240 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:51.613060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv"]
Sep 12 02:45:51.617904 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:45:51.617868    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f0e59c_c794_432d_a95d_de3483af1be3.slice/crio-b3e9987cc01ff100ebcf86a7624ec6a552ca37ff34f9dc400746c1386f3f14fc WatchSource:0}: Error finding container b3e9987cc01ff100ebcf86a7624ec6a552ca37ff34f9dc400746c1386f3f14fc: Status 404 returned error can't find the container with id b3e9987cc01ff100ebcf86a7624ec6a552ca37ff34f9dc400746c1386f3f14fc
Sep 12 02:45:52.573395 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:52.573352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv" event={"ID":"83f0e59c-c794-432d-a95d-de3483af1be3","Type":"ContainerStarted","Data":"b3e9987cc01ff100ebcf86a7624ec6a552ca37ff34f9dc400746c1386f3f14fc"}
Sep 12 02:45:54.585426 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:54.585387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv" event={"ID":"83f0e59c-c794-432d-a95d-de3483af1be3","Type":"ContainerStarted","Data":"a5e371860d32be9660512528297a1cd89e4d7f7361deb0cfcd68272aabf3852a"}
Sep 12 02:45:54.600048 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:45:54.599981    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84dbf54759-gw4sv" podStartSLOduration=1.730791713 podStartE2EDuration="4.599959931s" totalImagesPullingTime="2.869168218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:54.599240701 +0000 UTC m=+593.867274349" watchObservedRunningTime="2026-09-12 02:45:54.599959931 +0000 UTC m=+593.867993579"
Sep 12 02:47:24.767355 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.767321    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"]
Sep 12 02:47:24.770719 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.770697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"
Sep 12 02:47:24.773272 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.773241    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 02:47:24.773436 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.773324    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 02:47:24.773535 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.773507    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 12 02:47:24.773680 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.773662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n8r6z\""
Sep 12 02:47:24.783012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.782984    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"]
Sep 12 02:47:24.920443 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:24.920402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7vz\" (UniqueName: \"kubernetes.io/projected/f86ca843-b774-486d-9bbf-37a79b279890-kube-api-access-wl7vz\") pod \"dns-operator-controller-manager-7b84cd7c9c-799sf\" (UID: \"f86ca843-b774-486d-9bbf-37a79b279890\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"
Sep 12 02:47:25.021793 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:25.021700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7vz\" (UniqueName: \"kubernetes.io/projected/f86ca843-b774-486d-9bbf-37a79b279890-kube-api-access-wl7vz\") pod \"dns-operator-controller-manager-7b84cd7c9c-799sf\" (UID: \"f86ca843-b774-486d-9bbf-37a79b279890\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"
Sep 12 02:47:25.030078 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:25.030039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7vz\" (UniqueName: \"kubernetes.io/projected/f86ca843-b774-486d-9bbf-37a79b279890-kube-api-access-wl7vz\") pod \"dns-operator-controller-manager-7b84cd7c9c-799sf\" (UID: \"f86ca843-b774-486d-9bbf-37a79b279890\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"
Sep 12 02:47:25.084179 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:25.084140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"
Sep 12 02:47:25.360266 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:25.360186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"]
Sep 12 02:47:25.362706 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:47:25.362674    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86ca843_b774_486d_9bbf_37a79b279890.slice/crio-6e0909ef26e98751863699e65659db654dd61f3d0433895006c29b107b172120 WatchSource:0}: Error finding container 6e0909ef26e98751863699e65659db654dd61f3d0433895006c29b107b172120: Status 404 returned error can't find the container with id 6e0909ef26e98751863699e65659db654dd61f3d0433895006c29b107b172120
Sep 12 02:47:25.365705 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:25.365671    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:47:26.015054 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:26.015015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf" event={"ID":"f86ca843-b774-486d-9bbf-37a79b279890","Type":"ContainerStarted","Data":"6e0909ef26e98751863699e65659db654dd61f3d0433895006c29b107b172120"}
Sep 12 02:47:29.032377 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:29.032330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf" event={"ID":"f86ca843-b774-486d-9bbf-37a79b279890","Type":"ContainerStarted","Data":"4537c129a2d9f874e18c31f4c4386c0166c7a5a711b66e3ef7727aa0f54a012a"}
Sep 12 02:47:29.032789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:29.032412    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"
Sep 12 02:47:29.050710 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:29.050655    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf" podStartSLOduration=2.20396468 podStartE2EDuration="5.050637355s" totalImagesPullingTime="2.846672675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:29.049195646 +0000 UTC m=+688.317229306" watchObservedRunningTime="2026-09-12 02:47:29.050637355 +0000 UTC m=+688.318671002"
Sep 12 02:47:39.716724 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.716688    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"]
Sep 12 02:47:39.724783 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.724746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:39.727350 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.727325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjdc8\""
Sep 12 02:47:39.733878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.733535    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"]
Sep 12 02:47:39.754497 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.754459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrgx\" (UniqueName: \"kubernetes.io/projected/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-kube-api-access-jmrgx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:39.754680 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.754537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:39.855533 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.855487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrgx\" (UniqueName: \"kubernetes.io/projected/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-kube-api-access-jmrgx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:39.855710 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.855561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:39.855983 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.855963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:39.864448 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:39.864419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrgx\" (UniqueName: \"kubernetes.io/projected/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-kube-api-access-jmrgx\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:40.038226 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.038134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:40.039814 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.039793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-799sf"
Sep 12 02:47:40.330335 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.330238    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"]
Sep 12 02:47:40.334660 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:47:40.334631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f4cc76_83d2_4997_a97c_9cd4f9a11064.slice/crio-b84b888ffe7da3550952cd995c2a58cceaf08a229f8d0c1f4f8cca8a140c6d68 WatchSource:0}: Error finding container b84b888ffe7da3550952cd995c2a58cceaf08a229f8d0c1f4f8cca8a140c6d68: Status 404 returned error can't find the container with id b84b888ffe7da3550952cd995c2a58cceaf08a229f8d0c1f4f8cca8a140c6d68
Sep 12 02:47:40.778463 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.778420    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"]
Sep 12 02:47:40.791989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.791961    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"]
Sep 12 02:47:40.832967 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.832932    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"]
Sep 12 02:47:40.839157 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.839124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:40.858747 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.858715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"]
Sep 12 02:47:40.865666 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.865628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2cf95d-974b-452b-a85a-4a46f0ea5554-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mqptg\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:40.865835 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.865719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9nh\" (UniqueName: \"kubernetes.io/projected/6f2cf95d-974b-452b-a85a-4a46f0ea5554-kube-api-access-rd9nh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mqptg\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:40.966967 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.966915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2cf95d-974b-452b-a85a-4a46f0ea5554-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mqptg\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:40.967222 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.966984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9nh\" (UniqueName: \"kubernetes.io/projected/6f2cf95d-974b-452b-a85a-4a46f0ea5554-kube-api-access-rd9nh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mqptg\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:40.967435 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.967399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2cf95d-974b-452b-a85a-4a46f0ea5554-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mqptg\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:40.983487 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:40.983459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9nh\" (UniqueName: \"kubernetes.io/projected/6f2cf95d-974b-452b-a85a-4a46f0ea5554-kube-api-access-rd9nh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mqptg\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:41.151552 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:41.151451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:41.449721 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:41.449688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"]
Sep 12 02:47:41.453295 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:47:41.453261    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2cf95d_974b_452b_a85a_4a46f0ea5554.slice/crio-99991ff3f0be2e40195cdbee8bdc09aab5ccaa579f0e126623501401af17e81d WatchSource:0}: Error finding container 99991ff3f0be2e40195cdbee8bdc09aab5ccaa579f0e126623501401af17e81d: Status 404 returned error can't find the container with id 99991ff3f0be2e40195cdbee8bdc09aab5ccaa579f0e126623501401af17e81d
Sep 12 02:47:42.104412 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:42.104366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg" event={"ID":"6f2cf95d-974b-452b-a85a-4a46f0ea5554","Type":"ContainerStarted","Data":"99991ff3f0be2e40195cdbee8bdc09aab5ccaa579f0e126623501401af17e81d"}
Sep 12 02:47:45.125187 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.125135    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h" podUID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" containerName="manager" containerID="cri-o://d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3" gracePeriod=2
Sep 12 02:47:45.127156 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.127122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg" event={"ID":"6f2cf95d-974b-452b-a85a-4a46f0ea5554","Type":"ContainerStarted","Data":"e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70"}
Sep 12 02:47:45.127302 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.127278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:45.151140 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.151067    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg" podStartSLOduration=1.666277758 podStartE2EDuration="5.151049408s" totalImagesPullingTime="3.48477165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:45.148223211 +0000 UTC m=+704.416256859" watchObservedRunningTime="2026-09-12 02:47:45.151049408 +0000 UTC m=+704.419083054"
Sep 12 02:47:45.363468 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.363442    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:45.409893 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.409857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-extensions-socket-volume\") pod \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") "
Sep 12 02:47:45.410059 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.409959    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmrgx\" (UniqueName: \"kubernetes.io/projected/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-kube-api-access-jmrgx\") pod \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\" (UID: \"b3f4cc76-83d2-4997-a97c-9cd4f9a11064\") "
Sep 12 02:47:45.410201 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.410176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b3f4cc76-83d2-4997-a97c-9cd4f9a11064" (UID: "b3f4cc76-83d2-4997-a97c-9cd4f9a11064"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:47:45.412276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.412254    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-kube-api-access-jmrgx" (OuterVolumeSpecName: "kube-api-access-jmrgx") pod "b3f4cc76-83d2-4997-a97c-9cd4f9a11064" (UID: "b3f4cc76-83d2-4997-a97c-9cd4f9a11064"). InnerVolumeSpecName "kube-api-access-jmrgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:47:45.510949 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.510861    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrgx\" (UniqueName: \"kubernetes.io/projected/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-kube-api-access-jmrgx\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:47:45.510949 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:45.510891    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f4cc76-83d2-4997-a97c-9cd4f9a11064-extensions-socket-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:47:46.133587 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:46.133557    2579 generic.go:356] "Generic (PLEG): container finished" podID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" containerID="d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3" exitCode=2
Sep 12 02:47:46.134035 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:46.133602    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:46.134035 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:46.133655    2579 scope.go:117] "RemoveContainer" containerID="d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3"
Sep 12 02:47:46.136140 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:47:46.136079    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:46.143166 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:46.143142    2579 scope.go:117] "RemoveContainer" containerID="d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3"
Sep 12 02:47:46.143571 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:47:46.143540    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3\": container with ID starting with d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3 not found: ID does not exist" containerID="d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3"
Sep 12 02:47:46.143660 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:46.143584    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3"} err="failed to get container status \"d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3\": rpc error: code = NotFound desc = could not find container \"d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3\": container with ID starting with d1dc205db2306b7784530534049b467fceec33dc5b1b32b2dd97e9851572f7f3 not found: ID does not exist"
Sep 12 02:47:46.143713 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:47:46.143672    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-fpt2h\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-fpt2h"
Sep 12 02:47:47.250199 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:47.250166    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" path="/var/lib/kubelet/pods/b3f4cc76-83d2-4997-a97c-9cd4f9a11064/volumes"
Sep 12 02:47:52.789773 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:52.789691    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"]
Sep 12 02:47:52.790274 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:52.789919    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg" podUID="6f2cf95d-974b-452b-a85a-4a46f0ea5554" containerName="manager" containerID="cri-o://e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70" gracePeriod=10
Sep 12 02:47:52.791925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:52.791900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:53.051189 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.051110    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:53.140575 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.140533    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"]
Sep 12 02:47:53.141011 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.140991    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" containerName="manager"
Sep 12 02:47:53.141118 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.141012    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" containerName="manager"
Sep 12 02:47:53.141118 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.141033    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2cf95d-974b-452b-a85a-4a46f0ea5554" containerName="manager"
Sep 12 02:47:53.141118 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.141041    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2cf95d-974b-452b-a85a-4a46f0ea5554" containerName="manager"
Sep 12 02:47:53.141281 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.141157    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f4cc76-83d2-4997-a97c-9cd4f9a11064" containerName="manager"
Sep 12 02:47:53.141281 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.141175    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2cf95d-974b-452b-a85a-4a46f0ea5554" containerName="manager"
Sep 12 02:47:53.144072 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.144048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.157921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.157887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"]
Sep 12 02:47:53.171682 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.171644    2579 generic.go:356] "Generic (PLEG): container finished" podID="6f2cf95d-974b-452b-a85a-4a46f0ea5554" containerID="e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70" exitCode=0
Sep 12 02:47:53.171879 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.171695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"
Sep 12 02:47:53.171879 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.171702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg" event={"ID":"6f2cf95d-974b-452b-a85a-4a46f0ea5554","Type":"ContainerDied","Data":"e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70"}
Sep 12 02:47:53.171879 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.171753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg" event={"ID":"6f2cf95d-974b-452b-a85a-4a46f0ea5554","Type":"ContainerDied","Data":"99991ff3f0be2e40195cdbee8bdc09aab5ccaa579f0e126623501401af17e81d"}
Sep 12 02:47:53.171879 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.171776    2579 scope.go:117] "RemoveContainer" containerID="e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70"
Sep 12 02:47:53.180476 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.180456    2579 scope.go:117] "RemoveContainer" containerID="e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70"
Sep 12 02:47:53.180784 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:47:53.180758    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70\": container with ID starting with e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70 not found: ID does not exist" containerID="e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70"
Sep 12 02:47:53.180834 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.180787    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70"} err="failed to get container status \"e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70\": rpc error: code = NotFound desc = could not find container \"e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70\": container with ID starting with e1ef1cb643f18b6a8a6b9986b8bf289538253cf662c88a32f7f851d9badcbe70 not found: ID does not exist"
Sep 12 02:47:53.184309 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.184277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2cf95d-974b-452b-a85a-4a46f0ea5554-extensions-socket-volume\") pod \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") "
Sep 12 02:47:53.184453 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.184336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd9nh\" (UniqueName: \"kubernetes.io/projected/6f2cf95d-974b-452b-a85a-4a46f0ea5554-kube-api-access-rd9nh\") pod \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\" (UID: \"6f2cf95d-974b-452b-a85a-4a46f0ea5554\") "
Sep 12 02:47:53.185053 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.185025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2cf95d-974b-452b-a85a-4a46f0ea5554-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6f2cf95d-974b-452b-a85a-4a46f0ea5554" (UID: "6f2cf95d-974b-452b-a85a-4a46f0ea5554"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:47:53.186766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.186739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2cf95d-974b-452b-a85a-4a46f0ea5554-kube-api-access-rd9nh" (OuterVolumeSpecName: "kube-api-access-rd9nh") pod "6f2cf95d-974b-452b-a85a-4a46f0ea5554" (UID: "6f2cf95d-974b-452b-a85a-4a46f0ea5554"). InnerVolumeSpecName "kube-api-access-rd9nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:47:53.285541 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.285506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtc6\" (UniqueName: \"kubernetes.io/projected/aab37b6c-9946-4cc1-a864-de46312b11e3-kube-api-access-5wtc6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frlq9\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.285734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.285586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab37b6c-9946-4cc1-a864-de46312b11e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frlq9\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.285734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.285630    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2cf95d-974b-452b-a85a-4a46f0ea5554-extensions-socket-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:47:53.285734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.285641    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9nh\" (UniqueName: \"kubernetes.io/projected/6f2cf95d-974b-452b-a85a-4a46f0ea5554-kube-api-access-rd9nh\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:47:53.386499 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.386392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab37b6c-9946-4cc1-a864-de46312b11e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frlq9\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.386683 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.386560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtc6\" (UniqueName: \"kubernetes.io/projected/aab37b6c-9946-4cc1-a864-de46312b11e3-kube-api-access-5wtc6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frlq9\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.386800 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.386774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab37b6c-9946-4cc1-a864-de46312b11e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frlq9\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.395408 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.395368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtc6\" (UniqueName: \"kubernetes.io/projected/aab37b6c-9946-4cc1-a864-de46312b11e3-kube-api-access-5wtc6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frlq9\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.454950 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.454907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:53.492301 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.492247    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"]
Sep 12 02:47:53.500435 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.500403    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mqptg"]
Sep 12 02:47:53.745206 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:53.745000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"]
Sep 12 02:47:53.748377 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:47:53.748346    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab37b6c_9946_4cc1_a864_de46312b11e3.slice/crio-863f1a3f442bc44f384ba182d915b55f92b4471bfd1accd844cac1aeb70b4e6e WatchSource:0}: Error finding container 863f1a3f442bc44f384ba182d915b55f92b4471bfd1accd844cac1aeb70b4e6e: Status 404 returned error can't find the container with id 863f1a3f442bc44f384ba182d915b55f92b4471bfd1accd844cac1aeb70b4e6e
Sep 12 02:47:54.179513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:54.179418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9" event={"ID":"aab37b6c-9946-4cc1-a864-de46312b11e3","Type":"ContainerStarted","Data":"36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a"}
Sep 12 02:47:54.179513 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:54.179507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9" event={"ID":"aab37b6c-9946-4cc1-a864-de46312b11e3","Type":"ContainerStarted","Data":"863f1a3f442bc44f384ba182d915b55f92b4471bfd1accd844cac1aeb70b4e6e"}
Sep 12 02:47:54.179903 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:54.179626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:47:54.199774 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:54.199722    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9" podStartSLOduration=1.199704756 podStartE2EDuration="1.199704756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:54.198954219 +0000 UTC m=+713.466987866" watchObservedRunningTime="2026-09-12 02:47:54.199704756 +0000 UTC m=+713.467738404"
Sep 12 02:47:55.249647 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:47:55.249610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2cf95d-974b-452b-a85a-4a46f0ea5554" path="/var/lib/kubelet/pods/6f2cf95d-974b-452b-a85a-4a46f0ea5554/volumes"
Sep 12 02:48:05.187145 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:05.187086    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 02:48:26.252389 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.249630    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:48:26.255183 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.254027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.256918 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.256890    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 12 02:48:26.257063 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.256924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sq5df\""
Sep 12 02:48:26.261019 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.260989    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:48:26.283756 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.283717    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:48:26.285436 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.285411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1539d005-3280-49e1-b3f8-db9752ac3bf9-config-file\") pod \"limitador-limitador-84994f7ffc-9dt67\" (UID: \"1539d005-3280-49e1-b3f8-db9752ac3bf9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.285550 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.285460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bv8q\" (UniqueName: \"kubernetes.io/projected/1539d005-3280-49e1-b3f8-db9752ac3bf9-kube-api-access-5bv8q\") pod \"limitador-limitador-84994f7ffc-9dt67\" (UID: \"1539d005-3280-49e1-b3f8-db9752ac3bf9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.386816 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.386768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1539d005-3280-49e1-b3f8-db9752ac3bf9-config-file\") pod \"limitador-limitador-84994f7ffc-9dt67\" (UID: \"1539d005-3280-49e1-b3f8-db9752ac3bf9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.386816 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.386825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bv8q\" (UniqueName: \"kubernetes.io/projected/1539d005-3280-49e1-b3f8-db9752ac3bf9-kube-api-access-5bv8q\") pod \"limitador-limitador-84994f7ffc-9dt67\" (UID: \"1539d005-3280-49e1-b3f8-db9752ac3bf9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.387488 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.387466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1539d005-3280-49e1-b3f8-db9752ac3bf9-config-file\") pod \"limitador-limitador-84994f7ffc-9dt67\" (UID: \"1539d005-3280-49e1-b3f8-db9752ac3bf9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.394846 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.394815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bv8q\" (UniqueName: \"kubernetes.io/projected/1539d005-3280-49e1-b3f8-db9752ac3bf9-kube-api-access-5bv8q\") pod \"limitador-limitador-84994f7ffc-9dt67\" (UID: \"1539d005-3280-49e1-b3f8-db9752ac3bf9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.567838 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.567739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:26.842944 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:26.842904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:48:26.845350 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:48:26.845313    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1539d005_3280_49e1_b3f8_db9752ac3bf9.slice/crio-d4d005d62158ef339890bfffd4b5fa2bdc359afe678d2d90a4c5eba7d6a0208e WatchSource:0}: Error finding container d4d005d62158ef339890bfffd4b5fa2bdc359afe678d2d90a4c5eba7d6a0208e: Status 404 returned error can't find the container with id d4d005d62158ef339890bfffd4b5fa2bdc359afe678d2d90a4c5eba7d6a0208e
Sep 12 02:48:27.351929 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:27.351891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67" event={"ID":"1539d005-3280-49e1-b3f8-db9752ac3bf9","Type":"ContainerStarted","Data":"d4d005d62158ef339890bfffd4b5fa2bdc359afe678d2d90a4c5eba7d6a0208e"}
Sep 12 02:48:28.359213 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:28.359113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67" event={"ID":"1539d005-3280-49e1-b3f8-db9752ac3bf9","Type":"ContainerStarted","Data":"d5968f0bdf0ce8c998cf68345b8c5cd508bddf3ae1fc315a8abca2b0bd64787f"}
Sep 12 02:48:28.359578 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:28.359244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:28.375275 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:28.375212    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67" podStartSLOduration=1.286961254 podStartE2EDuration="2.375191511s" totalImagesPullingTime="1.088230257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:48:28.374283477 +0000 UTC m=+747.642317121" watchObservedRunningTime="2026-09-12 02:48:28.375191511 +0000 UTC m=+747.643225159"
Sep 12 02:48:39.366065 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:39.366033    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9dt67"
Sep 12 02:48:51.660796 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.660756    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bjm8x"]
Sep 12 02:48:51.664584 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.664560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:51.667891 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.667863    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5bgmr\""
Sep 12 02:48:51.667891 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.667895    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 12 02:48:51.670736 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.670709    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bjm8x"]
Sep 12 02:48:51.801105 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.801046    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bjm8x"]
Sep 12 02:48:51.801373 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:48:51.801350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p7nrv tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-bjm8x" podUID="48facaef-3a6b-4bec-b145-6c3ed84a0c13"
Sep 12 02:48:51.828770 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.828730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nrv\" (UniqueName: \"kubernetes.io/projected/48facaef-3a6b-4bec-b145-6c3ed84a0c13-kube-api-access-p7nrv\") pod \"authorino-6d4dfc649f-bjm8x\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") " pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:51.828958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.828781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48facaef-3a6b-4bec-b145-6c3ed84a0c13-tls-cert\") pod \"authorino-6d4dfc649f-bjm8x\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") " pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:51.929824 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.929790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48facaef-3a6b-4bec-b145-6c3ed84a0c13-tls-cert\") pod \"authorino-6d4dfc649f-bjm8x\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") " pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:51.929987 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.929893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nrv\" (UniqueName: \"kubernetes.io/projected/48facaef-3a6b-4bec-b145-6c3ed84a0c13-kube-api-access-p7nrv\") pod \"authorino-6d4dfc649f-bjm8x\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") " pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:51.932609 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.932585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48facaef-3a6b-4bec-b145-6c3ed84a0c13-tls-cert\") pod \"authorino-6d4dfc649f-bjm8x\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") " pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:51.937874 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:51.937838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nrv\" (UniqueName: \"kubernetes.io/projected/48facaef-3a6b-4bec-b145-6c3ed84a0c13-kube-api-access-p7nrv\") pod \"authorino-6d4dfc649f-bjm8x\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") " pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:52.475347 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.475315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:52.482889 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.482858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:52.635562 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.635518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7nrv\" (UniqueName: \"kubernetes.io/projected/48facaef-3a6b-4bec-b145-6c3ed84a0c13-kube-api-access-p7nrv\") pod \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") "
Sep 12 02:48:52.635562 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.635568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48facaef-3a6b-4bec-b145-6c3ed84a0c13-tls-cert\") pod \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\" (UID: \"48facaef-3a6b-4bec-b145-6c3ed84a0c13\") "
Sep 12 02:48:52.637986 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.637945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48facaef-3a6b-4bec-b145-6c3ed84a0c13-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "48facaef-3a6b-4bec-b145-6c3ed84a0c13" (UID: "48facaef-3a6b-4bec-b145-6c3ed84a0c13"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:48:52.637986 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.637967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48facaef-3a6b-4bec-b145-6c3ed84a0c13-kube-api-access-p7nrv" (OuterVolumeSpecName: "kube-api-access-p7nrv") pod "48facaef-3a6b-4bec-b145-6c3ed84a0c13" (UID: "48facaef-3a6b-4bec-b145-6c3ed84a0c13"). InnerVolumeSpecName "kube-api-access-p7nrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:48:52.736718 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.736616    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7nrv\" (UniqueName: \"kubernetes.io/projected/48facaef-3a6b-4bec-b145-6c3ed84a0c13-kube-api-access-p7nrv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:48:52.736718 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:52.736654    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/48facaef-3a6b-4bec-b145-6c3ed84a0c13-tls-cert\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:48:53.479808 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:53.479772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bjm8x"
Sep 12 02:48:53.504952 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:53.504846    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bjm8x"]
Sep 12 02:48:53.506650 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:53.506621    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bjm8x"]
Sep 12 02:48:55.249223 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:48:55.249195    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48facaef-3a6b-4bec-b145-6c3ed84a0c13" path="/var/lib/kubelet/pods/48facaef-3a6b-4bec-b145-6c3ed84a0c13/volumes"
Sep 12 02:49:34.387505 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:49:34.387418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:49:55.256285 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:49:55.256253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:08.014195 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.014151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"]
Sep 12 02:50:08.017616 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.017593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.020111 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.020071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 12 02:50:08.020246 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.020111    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 12 02:50:08.021272 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.021256    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wxtgp\""
Sep 12 02:50:08.021385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.021256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 12 02:50:08.026862 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.026833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"]
Sep 12 02:50:08.159187 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.159141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.159187 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.159188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.159453 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.159223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlc6\" (UniqueName: \"kubernetes.io/projected/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-kube-api-access-5qlc6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.159453 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.159314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.159453 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.159362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.159453 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.159384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.260554 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.260509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.260554 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.260558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.260831 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.260621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.260831 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.260637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.260831 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.260656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlc6\" (UniqueName: \"kubernetes.io/projected/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-kube-api-access-5qlc6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.260831 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.260719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.261146 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.261076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.261267 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.261151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.261267 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.261160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.263160 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.263130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.263406 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.263386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.269493 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.269422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlc6\" (UniqueName: \"kubernetes.io/projected/acd0dcd8-7df4-414b-ab2c-06cef1a94c6f-kube-api-access-5qlc6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-scg64\" (UID: \"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.329298 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.329251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:08.608817 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.608769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"]
Sep 12 02:50:08.611834 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:50:08.611794    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd0dcd8_7df4_414b_ab2c_06cef1a94c6f.slice/crio-1f8fbb5cbc274ebe304f23aea00a7727330c9372163e41bb0d80be31904df6d5 WatchSource:0}: Error finding container 1f8fbb5cbc274ebe304f23aea00a7727330c9372163e41bb0d80be31904df6d5: Status 404 returned error can't find the container with id 1f8fbb5cbc274ebe304f23aea00a7727330c9372163e41bb0d80be31904df6d5
Sep 12 02:50:08.856983 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.856943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64" event={"ID":"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f","Type":"ContainerStarted","Data":"1f8fbb5cbc274ebe304f23aea00a7727330c9372163e41bb0d80be31904df6d5"}
Sep 12 02:50:08.943693 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:08.943640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:10.868573 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:10.868528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64" event={"ID":"acd0dcd8-7df4-414b-ab2c-06cef1a94c6f","Type":"ContainerStarted","Data":"9235981ac38cd231cc3f9b132afaa365838c40a48c63f0d64bfd0df7d49e0e09"}
Sep 12 02:50:10.885396 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:10.885321    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64" podStartSLOduration=2.618392632 podStartE2EDuration="3.88530202s" totalImagesPullingTime="1.266909388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:50:10.884040722 +0000 UTC m=+850.152074369" watchObservedRunningTime="2026-09-12 02:50:10.88530202 +0000 UTC m=+850.153335667"
Sep 12 02:50:13.242703 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:13.242654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:18.330313 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:18.330272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:18.330841 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:18.330359    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:18.343215 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:18.343186    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:18.647468 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:18.647377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:18.921301 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:18.921207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-scg64"
Sep 12 02:50:30.745015 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:30.744977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:37.344623 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:37.344583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:44.412555 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.412519    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"]
Sep 12 02:50:44.418539 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.418510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.421264 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.421236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 12 02:50:44.424909 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.424885    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"]
Sep 12 02:50:44.501196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.501158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.501398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.501221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.501398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.501244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.501398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.501314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.501398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.501344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzjg\" (UniqueName: \"kubernetes.io/projected/de080106-f89d-4e28-8754-a85c0e8baef4-kube-api-access-8tzjg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.501398 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.501379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de080106-f89d-4e28-8754-a85c0e8baef4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.602555 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.602516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.602761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.602574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.602761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.602597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzjg\" (UniqueName: \"kubernetes.io/projected/de080106-f89d-4e28-8754-a85c0e8baef4-kube-api-access-8tzjg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.602761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.602625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de080106-f89d-4e28-8754-a85c0e8baef4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.602761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.602691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.602761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.602731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.603016 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.602998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.603135 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.603116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.603197 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.603177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.605433 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.605406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de080106-f89d-4e28-8754-a85c0e8baef4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.605540 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.605444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de080106-f89d-4e28-8754-a85c0e8baef4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.610684 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.610655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzjg\" (UniqueName: \"kubernetes.io/projected/de080106-f89d-4e28-8754-a85c0e8baef4-kube-api-access-8tzjg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2\" (UID: \"de080106-f89d-4e28-8754-a85c0e8baef4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:44.730237 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:44.730198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:45.006602 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:45.006571    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"]
Sep 12 02:50:45.009401 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:50:45.009376    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde080106_f89d_4e28_8754_a85c0e8baef4.slice/crio-804498c4ddaf621547004d25d64e23d0f35c353c03ee32f3ec40ef19e341f7e0 WatchSource:0}: Error finding container 804498c4ddaf621547004d25d64e23d0f35c353c03ee32f3ec40ef19e341f7e0: Status 404 returned error can't find the container with id 804498c4ddaf621547004d25d64e23d0f35c353c03ee32f3ec40ef19e341f7e0
Sep 12 02:50:45.039062 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:45.039030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2" event={"ID":"de080106-f89d-4e28-8754-a85c0e8baef4","Type":"ContainerStarted","Data":"804498c4ddaf621547004d25d64e23d0f35c353c03ee32f3ec40ef19e341f7e0"}
Sep 12 02:50:45.941448 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:45.941412    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:46.045629 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:46.045588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2" event={"ID":"de080106-f89d-4e28-8754-a85c0e8baef4","Type":"ContainerStarted","Data":"f9098994d4155242a5322c54f043a275650988fc322b748c80696dc15a584aea"}
Sep 12 02:50:46.061968 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:46.061915    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2" podStartSLOduration=1.877944507 podStartE2EDuration="2.061898375s" totalImagesPullingTime="183.953868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:50:46.061054081 +0000 UTC m=+885.329087730" watchObservedRunningTime="2026-09-12 02:50:46.061898375 +0000 UTC m=+885.329932040"
Sep 12 02:50:52.643634 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:52.643544    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:50:54.730722 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:54.730689    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:54.731288 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:54.730845    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:54.743308 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:54.743283    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:50:55.102486 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:50:55.102399    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2"
Sep 12 02:52:21.141010 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.140977    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7891-2bdadb"]
Sep 12 02:52:21.144497 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.144481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-2bdadb"
Sep 12 02:52:21.146854 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.146829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sww9\""
Sep 12 02:52:21.150925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.150903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-2bdadb"]
Sep 12 02:52:21.245438 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.245405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq59\" (UniqueName: \"kubernetes.io/projected/d315d47d-6f8c-426b-ad61-4061c7ce9962-kube-api-access-rnq59\") pod \"test-curl-7891-2bdadb\" (UID: \"d315d47d-6f8c-426b-ad61-4061c7ce9962\") " pod="opendatahub/test-curl-7891-2bdadb"
Sep 12 02:52:21.346410 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.346378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq59\" (UniqueName: \"kubernetes.io/projected/d315d47d-6f8c-426b-ad61-4061c7ce9962-kube-api-access-rnq59\") pod \"test-curl-7891-2bdadb\" (UID: \"d315d47d-6f8c-426b-ad61-4061c7ce9962\") " pod="opendatahub/test-curl-7891-2bdadb"
Sep 12 02:52:21.354142 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.354119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq59\" (UniqueName: \"kubernetes.io/projected/d315d47d-6f8c-426b-ad61-4061c7ce9962-kube-api-access-rnq59\") pod \"test-curl-7891-2bdadb\" (UID: \"d315d47d-6f8c-426b-ad61-4061c7ce9962\") " pod="opendatahub/test-curl-7891-2bdadb"
Sep 12 02:52:21.454985 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.454957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-2bdadb"
Sep 12 02:52:21.726240 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:21.726218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-2bdadb"]
Sep 12 02:52:21.729049 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:52:21.729023    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd315d47d_6f8c_426b_ad61_4061c7ce9962.slice/crio-b9c40636384e5df5396afe1ee860de2b95283d577ecb18dac27d0b7b8d486b79 WatchSource:0}: Error finding container b9c40636384e5df5396afe1ee860de2b95283d577ecb18dac27d0b7b8d486b79: Status 404 returned error can't find the container with id b9c40636384e5df5396afe1ee860de2b95283d577ecb18dac27d0b7b8d486b79
Sep 12 02:52:22.518453 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:22.518406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-2bdadb" event={"ID":"d315d47d-6f8c-426b-ad61-4061c7ce9962","Type":"ContainerStarted","Data":"b9c40636384e5df5396afe1ee860de2b95283d577ecb18dac27d0b7b8d486b79"}
Sep 12 02:52:22.553960 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:22.553920    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:52:24.529442 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:24.529409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-2bdadb" event={"ID":"d315d47d-6f8c-426b-ad61-4061c7ce9962","Type":"ContainerStarted","Data":"d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80"}
Sep 12 02:52:24.545161 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:24.545073    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7891-2bdadb" podStartSLOduration=1.488985514 podStartE2EDuration="3.545055109s" totalImagesPullingTime="2.056069595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:24.544587081 +0000 UTC m=+983.812620729" watchObservedRunningTime="2026-09-12 02:52:24.545055109 +0000 UTC m=+983.813088756"
Sep 12 02:52:25.178829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.178799    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7891-2bdadb"]
Sep 12 02:52:25.192005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.191944    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7891-2bdadb"]
Sep 12 02:52:25.314557 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.314526    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7891-152eb1"]
Sep 12 02:52:25.315033 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.315011    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d315d47d-6f8c-426b-ad61-4061c7ce9962" containerName="test-curl-7891-2bdadb"
Sep 12 02:52:25.315033 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.315035    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d315d47d-6f8c-426b-ad61-4061c7ce9962" containerName="test-curl-7891-2bdadb"
Sep 12 02:52:25.315238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.315157    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d315d47d-6f8c-426b-ad61-4061c7ce9962" containerName="test-curl-7891-2bdadb"
Sep 12 02:52:25.318361 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.318342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-152eb1"
Sep 12 02:52:25.322043 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.322017    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-152eb1"]
Sep 12 02:52:25.384018 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.383987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5hs\" (UniqueName: \"kubernetes.io/projected/6a79e68d-6511-476e-a34c-dbf7f80c28cb-kube-api-access-fn5hs\") pod \"test-curl-7891-152eb1\" (UID: \"6a79e68d-6511-476e-a34c-dbf7f80c28cb\") " pod="opendatahub/test-curl-7891-152eb1"
Sep 12 02:52:25.485212 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.485129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5hs\" (UniqueName: \"kubernetes.io/projected/6a79e68d-6511-476e-a34c-dbf7f80c28cb-kube-api-access-fn5hs\") pod \"test-curl-7891-152eb1\" (UID: \"6a79e68d-6511-476e-a34c-dbf7f80c28cb\") " pod="opendatahub/test-curl-7891-152eb1"
Sep 12 02:52:25.494966 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.494936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5hs\" (UniqueName: \"kubernetes.io/projected/6a79e68d-6511-476e-a34c-dbf7f80c28cb-kube-api-access-fn5hs\") pod \"test-curl-7891-152eb1\" (UID: \"6a79e68d-6511-476e-a34c-dbf7f80c28cb\") " pod="opendatahub/test-curl-7891-152eb1"
Sep 12 02:52:25.533810 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:25.533785    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7891-2bdadb" container="test-curl-7891-2bdadb"
Sep 12 02:52:25.628468 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.628443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-152eb1"
Sep 12 02:52:25.719591 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.719523    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"]
Sep 12 02:52:25.723807 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.723788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.726669 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.726646    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c6190b-serving-cert\""
Sep 12 02:52:25.726995 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.726981    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 02:52:25.727582 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.727563    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 02:52:25.727676 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.727563    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 02:52:25.727676 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.727609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsxrv\""
Sep 12 02:52:25.734284 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.734264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"]
Sep 12 02:52:25.889298 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.889219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsc8\" (UniqueName: \"kubernetes.io/projected/451557bb-4bca-4342-b716-e1c95b08545c-kube-api-access-6gsc8\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.889298 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.889287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.889494 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.889309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451557bb-4bca-4342-b716-e1c95b08545c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.889494 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.889327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-metrics-certs\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.899296 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.899275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7891-152eb1"]
Sep 12 02:52:25.901825 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:52:25.901801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a79e68d_6511_476e_a34c_dbf7f80c28cb.slice/crio-c5a492db762e3ee3c16e85c07610a68e02a8cb488a9f1e6d9770162408d0157f WatchSource:0}: Error finding container c5a492db762e3ee3c16e85c07610a68e02a8cb488a9f1e6d9770162408d0157f: Status 404 returned error can't find the container with id c5a492db762e3ee3c16e85c07610a68e02a8cb488a9f1e6d9770162408d0157f
Sep 12 02:52:25.990632 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.990600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-metrics-certs\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.990732 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.990675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsc8\" (UniqueName: \"kubernetes.io/projected/451557bb-4bca-4342-b716-e1c95b08545c-kube-api-access-6gsc8\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.990796 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.990781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.990836 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.990819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451557bb-4bca-4342-b716-e1c95b08545c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.991580 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.991553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451557bb-4bca-4342-b716-e1c95b08545c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.992940 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.992921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-metrics-certs\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.993027 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.993009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:25.997657 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:25.997634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsc8\" (UniqueName: \"kubernetes.io/projected/451557bb-4bca-4342-b716-e1c95b08545c-kube-api-access-6gsc8\") pod \"maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:26.038008 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.037984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:26.310871 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.310847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"]
Sep 12 02:52:26.313702 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:52:26.313675    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451557bb_4bca_4342_b716_e1c95b08545c.slice/crio-88ec991070dc96d77772e3fe26f56595ce95fe77d7e4c007511459bf5b095eee WatchSource:0}: Error finding container 88ec991070dc96d77772e3fe26f56595ce95fe77d7e4c007511459bf5b095eee: Status 404 returned error can't find the container with id 88ec991070dc96d77772e3fe26f56595ce95fe77d7e4c007511459bf5b095eee
Sep 12 02:52:26.315184 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.315165    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:52:26.539276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.539233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2" event={"ID":"451557bb-4bca-4342-b716-e1c95b08545c","Type":"ContainerStarted","Data":"88ec991070dc96d77772e3fe26f56595ce95fe77d7e4c007511459bf5b095eee"}
Sep 12 02:52:26.540720 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.540694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-152eb1" event={"ID":"6a79e68d-6511-476e-a34c-dbf7f80c28cb","Type":"ContainerStarted","Data":"aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8"}
Sep 12 02:52:26.540865 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.540729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7891-152eb1" event={"ID":"6a79e68d-6511-476e-a34c-dbf7f80c28cb","Type":"ContainerStarted","Data":"c5a492db762e3ee3c16e85c07610a68e02a8cb488a9f1e6d9770162408d0157f"}
Sep 12 02:52:26.540865 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.540753    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7891-2bdadb" podUID="d315d47d-6f8c-426b-ad61-4061c7ce9962" containerName="test-curl-7891-2bdadb" containerID="cri-o://d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80" gracePeriod=2
Sep 12 02:52:26.554079 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:26.554037    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7891-152eb1" podStartSLOduration=1.5540218270000001 podStartE2EDuration="1.554021827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:26.552679977 +0000 UTC m=+985.820713636" watchObservedRunningTime="2026-09-12 02:52:26.554021827 +0000 UTC m=+985.822055474"
Sep 12 02:52:27.173030 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:27.172911    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7891-152eb1"]
Sep 12 02:52:27.178915 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:27.178856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7891-152eb1"]
Sep 12 02:52:27.547901 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:27.547873    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7891-152eb1" container="test-curl-7891-152eb1"
Sep 12 02:52:28.554068 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:28.553914    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7891-152eb1" podUID="6a79e68d-6511-476e-a34c-dbf7f80c28cb" containerName="test-curl-7891-152eb1" containerID="cri-o://aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8" gracePeriod=2
Sep 12 02:52:28.992086 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:28.992060    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-2bdadb"
Sep 12 02:52:29.122463 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.122368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnq59\" (UniqueName: \"kubernetes.io/projected/d315d47d-6f8c-426b-ad61-4061c7ce9962-kube-api-access-rnq59\") pod \"d315d47d-6f8c-426b-ad61-4061c7ce9962\" (UID: \"d315d47d-6f8c-426b-ad61-4061c7ce9962\") "
Sep 12 02:52:29.125219 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.125182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d315d47d-6f8c-426b-ad61-4061c7ce9962-kube-api-access-rnq59" (OuterVolumeSpecName: "kube-api-access-rnq59") pod "d315d47d-6f8c-426b-ad61-4061c7ce9962" (UID: "d315d47d-6f8c-426b-ad61-4061c7ce9962"). InnerVolumeSpecName "kube-api-access-rnq59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:52:29.223568 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.223532    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnq59\" (UniqueName: \"kubernetes.io/projected/d315d47d-6f8c-426b-ad61-4061c7ce9962-kube-api-access-rnq59\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:52:29.250250 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.250223    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d315d47d-6f8c-426b-ad61-4061c7ce9962" path="/var/lib/kubelet/pods/d315d47d-6f8c-426b-ad61-4061c7ce9962/volumes"
Sep 12 02:52:29.559807 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.559775    2579 generic.go:356] "Generic (PLEG): container finished" podID="d315d47d-6f8c-426b-ad61-4061c7ce9962" containerID="d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80" exitCode=137
Sep 12 02:52:29.560296 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.559826    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-2bdadb"
Sep 12 02:52:29.560296 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.559889    2579 scope.go:117] "RemoveContainer" containerID="d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80"
Sep 12 02:52:29.561802 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.561772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2" event={"ID":"451557bb-4bca-4342-b716-e1c95b08545c","Type":"ContainerStarted","Data":"e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d"}
Sep 12 02:52:29.561978 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.561956    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:29.568577 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.568560    2579 scope.go:117] "RemoveContainer" containerID="d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80"
Sep 12 02:52:29.568880 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:29.568858    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80\": container with ID starting with d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80 not found: ID does not exist" containerID="d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80"
Sep 12 02:52:29.568928 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.568890    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80"} err="failed to get container status \"d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80\": rpc error: code = NotFound desc = could not find container \"d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80\": container with ID starting with d8e5b7c674f931252ae492133762102320292bcc411cf90f52be70a59f370b80 not found: ID does not exist"
Sep 12 02:52:29.579742 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:29.579699    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2" podStartSLOduration=1.993683972 podStartE2EDuration="4.579685369s" totalImagesPullingTime="2.586001397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:29.57765744 +0000 UTC m=+988.845691089" watchObservedRunningTime="2026-09-12 02:52:29.579685369 +0000 UTC m=+988.847719016"
Sep 12 02:52:30.713379 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:30.713355    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-152eb1"
Sep 12 02:52:30.839724 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:30.839663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn5hs\" (UniqueName: \"kubernetes.io/projected/6a79e68d-6511-476e-a34c-dbf7f80c28cb-kube-api-access-fn5hs\") pod \"6a79e68d-6511-476e-a34c-dbf7f80c28cb\" (UID: \"6a79e68d-6511-476e-a34c-dbf7f80c28cb\") "
Sep 12 02:52:30.841765 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:30.841746    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a79e68d-6511-476e-a34c-dbf7f80c28cb-kube-api-access-fn5hs" (OuterVolumeSpecName: "kube-api-access-fn5hs") pod "6a79e68d-6511-476e-a34c-dbf7f80c28cb" (UID: "6a79e68d-6511-476e-a34c-dbf7f80c28cb"). InnerVolumeSpecName "kube-api-access-fn5hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:52:30.940349 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:30.940324    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5hs\" (UniqueName: \"kubernetes.io/projected/6a79e68d-6511-476e-a34c-dbf7f80c28cb-kube-api-access-fn5hs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:52:31.248954 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:31.248922    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a79e68d-6511-476e-a34c-dbf7f80c28cb" path="/var/lib/kubelet/pods/6a79e68d-6511-476e-a34c-dbf7f80c28cb/volumes"
Sep 12 02:52:31.574452 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:31.574357    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a79e68d-6511-476e-a34c-dbf7f80c28cb" containerID="aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8" exitCode=137
Sep 12 02:52:31.574452 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:31.574398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7891-152eb1"
Sep 12 02:52:31.574638 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:31.574475    2579 scope.go:117] "RemoveContainer" containerID="aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8"
Sep 12 02:52:31.583653 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:31.583622    2579 scope.go:117] "RemoveContainer" containerID="aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8"
Sep 12 02:52:31.583954 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:31.583934    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8\": container with ID starting with aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8 not found: ID does not exist" containerID="aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8"
Sep 12 02:52:31.584030 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:31.583961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8"} err="failed to get container status \"aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8\": rpc error: code = NotFound desc = could not find container \"aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8\": container with ID starting with aa7fab2af0fe9c1d1b08353693384a89138adb164f721b52b5aace4c70188bb8 not found: ID does not exist"
Sep 12 02:52:34.133796 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.133756    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"]
Sep 12 02:52:34.134378 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.134358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a79e68d-6511-476e-a34c-dbf7f80c28cb" containerName="test-curl-7891-152eb1"
Sep 12 02:52:34.134378 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.134379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a79e68d-6511-476e-a34c-dbf7f80c28cb" containerName="test-curl-7891-152eb1"
Sep 12 02:52:34.134525 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.134471    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a79e68d-6511-476e-a34c-dbf7f80c28cb" containerName="test-curl-7891-152eb1"
Sep 12 02:52:34.139225 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.139200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.141365 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.141345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-214c0e42-serving-cert\""
Sep 12 02:52:34.147116 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.147066    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"]
Sep 12 02:52:34.267952 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.267919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5164d2-9413-460a-ab29-fc1521076c1a-openshift-service-ca\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.268162 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.267969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdzm\" (UniqueName: \"kubernetes.io/projected/ea5164d2-9413-460a-ab29-fc1521076c1a-kube-api-access-wcdzm\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.268162 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.267998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-metrics-certs\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.268162 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.268017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-maas-api-tls\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.369082 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.369051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5164d2-9413-460a-ab29-fc1521076c1a-openshift-service-ca\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.369260 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.369116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdzm\" (UniqueName: \"kubernetes.io/projected/ea5164d2-9413-460a-ab29-fc1521076c1a-kube-api-access-wcdzm\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.369260 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.369151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-metrics-certs\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.369260 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.369171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-maas-api-tls\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.369805 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.369782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5164d2-9413-460a-ab29-fc1521076c1a-openshift-service-ca\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.371758 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.371727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-metrics-certs\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.371847 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.371760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-maas-api-tls\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.376637 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.376613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdzm\" (UniqueName: \"kubernetes.io/projected/ea5164d2-9413-460a-ab29-fc1521076c1a-kube-api-access-wcdzm\") pod \"maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.451235 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.451209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:34.733573 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:34.733545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"]
Sep 12 02:52:34.736891 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:52:34.736860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5164d2_9413_460a_ab29_fc1521076c1a.slice/crio-5cbd105ece35440d81d458d6f5e7c9b36e599418a30a6119179960da0d26fc0f WatchSource:0}: Error finding container 5cbd105ece35440d81d458d6f5e7c9b36e599418a30a6119179960da0d26fc0f: Status 404 returned error can't find the container with id 5cbd105ece35440d81d458d6f5e7c9b36e599418a30a6119179960da0d26fc0f
Sep 12 02:52:35.045027 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.044948    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"]
Sep 12 02:52:35.054687 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.054658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.059837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.057646    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4b3be7-serving-cert\""
Sep 12 02:52:35.059837 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.058939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"]
Sep 12 02:52:35.176651 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.176626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcrnk\" (UniqueName: \"kubernetes.io/projected/98a1a921-9247-4e57-b27b-b4468b03ce42-kube-api-access-vcrnk\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.176939 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.176670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1a921-9247-4e57-b27b-b4468b03ce42-openshift-service-ca\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.176939 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.176746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.176939 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.176784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-metrics-certs\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.277657 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.277626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcrnk\" (UniqueName: \"kubernetes.io/projected/98a1a921-9247-4e57-b27b-b4468b03ce42-kube-api-access-vcrnk\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.277812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.277673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1a921-9247-4e57-b27b-b4468b03ce42-openshift-service-ca\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.277812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.277723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.277812 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.277756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-metrics-certs\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.277918 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:35.277868    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-serving-cert: secret "maas-api-e2e-autores-4b3be7-serving-cert" not found
Sep 12 02:52:35.277955 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:35.277933    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls podName:98a1a921-9247-4e57-b27b-b4468b03ce42 nodeName:}" failed. No retries permitted until 2026-09-12 02:52:35.777914713 +0000 UTC m=+995.045948338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls") pod "maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z" (UID: "98a1a921-9247-4e57-b27b-b4468b03ce42") : secret "maas-api-e2e-autores-4b3be7-serving-cert" not found
Sep 12 02:52:35.278539 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.278480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1a921-9247-4e57-b27b-b4468b03ce42-openshift-service-ca\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.280299 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.280278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-metrics-certs\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.285632 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.285607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcrnk\" (UniqueName: \"kubernetes.io/projected/98a1a921-9247-4e57-b27b-b4468b03ce42-kube-api-access-vcrnk\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.574793 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.574766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 02:52:35.597393 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.597358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9" event={"ID":"ea5164d2-9413-460a-ab29-fc1521076c1a","Type":"ContainerStarted","Data":"1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de"}
Sep 12 02:52:35.597561 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.597408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9" event={"ID":"ea5164d2-9413-460a-ab29-fc1521076c1a","Type":"ContainerStarted","Data":"5cbd105ece35440d81d458d6f5e7c9b36e599418a30a6119179960da0d26fc0f"}
Sep 12 02:52:35.597561 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.597499    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:35.616717 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.616661    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9" podStartSLOduration=1.202680268 podStartE2EDuration="1.616641183s" totalImagesPullingTime="413.960915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:35.616476745 +0000 UTC m=+994.884510404" watchObservedRunningTime="2026-09-12 02:52:35.616641183 +0000 UTC m=+994.884674831"
Sep 12 02:52:35.782278 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.782241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.784721 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.784700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls\") pod \"maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:35.970342 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:35.970306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:36.243792 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:36.243767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"]
Sep 12 02:52:36.245439 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:52:36.245409    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a1a921_9247_4e57_b27b_b4468b03ce42.slice/crio-bcb39aa91037b950c6eb073abae5d94f5aaf76d8f25f86802aafb72a5c4b5167 WatchSource:0}: Error finding container bcb39aa91037b950c6eb073abae5d94f5aaf76d8f25f86802aafb72a5c4b5167: Status 404 returned error can't find the container with id bcb39aa91037b950c6eb073abae5d94f5aaf76d8f25f86802aafb72a5c4b5167
Sep 12 02:52:36.603829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:36.603753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z" event={"ID":"98a1a921-9247-4e57-b27b-b4468b03ce42","Type":"ContainerStarted","Data":"bcb39aa91037b950c6eb073abae5d94f5aaf76d8f25f86802aafb72a5c4b5167"}
Sep 12 02:52:37.611147 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:37.611107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z" event={"ID":"98a1a921-9247-4e57-b27b-b4468b03ce42","Type":"ContainerStarted","Data":"4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0"}
Sep 12 02:52:37.611550 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:37.611230    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:37.631586 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:37.631537    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z" podStartSLOduration=2.25033007 podStartE2EDuration="2.631521647s" totalImagesPullingTime="381.191577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:37.628963581 +0000 UTC m=+996.896997227" watchObservedRunningTime="2026-09-12 02:52:37.631521647 +0000 UTC m=+996.899555290"
Sep 12 02:52:39.154276 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:39.154240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:52:41.609427 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:41.609395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:52:43.621977 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:43.621944    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:52:50.541388 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.541349    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"]
Sep 12 02:52:50.548529 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.548506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.550788 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.550767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f93121-serving-cert\""
Sep 12 02:52:50.556196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.556172    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"]
Sep 12 02:52:50.621539 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.621505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4d163d-b8cc-49db-93d9-981928dfcea2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.621539 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.621540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68llr\" (UniqueName: \"kubernetes.io/projected/bb4d163d-b8cc-49db-93d9-981928dfcea2-kube-api-access-68llr\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.621693 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.621630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-metrics-certs\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.621741 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.621713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.722976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.722943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.723114 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.722981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4d163d-b8cc-49db-93d9-981928dfcea2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.723114 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.722997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68llr\" (UniqueName: \"kubernetes.io/projected/bb4d163d-b8cc-49db-93d9-981928dfcea2-kube-api-access-68llr\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.723114 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.723067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-metrics-certs\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.723242 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:50.723084    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-serving-cert: secret "maas-api-e2e-inf-b-f93121-serving-cert" not found
Sep 12 02:52:50.723242 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:52:50.723192    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls podName:bb4d163d-b8cc-49db-93d9-981928dfcea2 nodeName:}" failed. No retries permitted until 2026-09-12 02:52:51.223175258 +0000 UTC m=+1010.491208887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls") pod "maas-api-e2e-inf-b-f93121-86998b4774-k2d7w" (UID: "bb4d163d-b8cc-49db-93d9-981928dfcea2") : secret "maas-api-e2e-inf-b-f93121-serving-cert" not found
Sep 12 02:52:50.723750 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.723727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4d163d-b8cc-49db-93d9-981928dfcea2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.725783 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.725762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-metrics-certs\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:50.732428 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:50.732409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68llr\" (UniqueName: \"kubernetes.io/projected/bb4d163d-b8cc-49db-93d9-981928dfcea2-kube-api-access-68llr\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:51.228493 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:51.228459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:51.231085 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:51.231063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls\") pod \"maas-api-e2e-inf-b-f93121-86998b4774-k2d7w\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:51.464544 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:51.464513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:51.737832 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:51.737803    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"]
Sep 12 02:52:51.740408 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:52:51.740383    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4d163d_b8cc_49db_93d9_981928dfcea2.slice/crio-bbee99dba2412446804d9649b62ff74d978bdaed27307a4016aaea47cd5c9c70 WatchSource:0}: Error finding container bbee99dba2412446804d9649b62ff74d978bdaed27307a4016aaea47cd5c9c70: Status 404 returned error can't find the container with id bbee99dba2412446804d9649b62ff74d978bdaed27307a4016aaea47cd5c9c70
Sep 12 02:52:52.685363 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:52.685334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w" event={"ID":"bb4d163d-b8cc-49db-93d9-981928dfcea2","Type":"ContainerStarted","Data":"fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca"}
Sep 12 02:52:52.685598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:52.685383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w" event={"ID":"bb4d163d-b8cc-49db-93d9-981928dfcea2","Type":"ContainerStarted","Data":"bbee99dba2412446804d9649b62ff74d978bdaed27307a4016aaea47cd5c9c70"}
Sep 12 02:52:52.685598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:52.685443    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:52:52.703144 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:52.703086    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w" podStartSLOduration=2.254816168 podStartE2EDuration="2.703071768s" totalImagesPullingTime="448.2556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:52.700810398 +0000 UTC m=+1011.968844045" watchObservedRunningTime="2026-09-12 02:52:52.703071768 +0000 UTC m=+1011.971105415"
Sep 12 02:52:55.354390 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:55.354355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:52:58.696336 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:52:58.696308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 02:53:02.187668 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.187636    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"]
Sep 12 02:53:02.188073 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.187865    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9" podUID="ea5164d2-9413-460a-ab29-fc1521076c1a" containerName="maas-api" containerID="cri-o://1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de" gracePeriod=30
Sep 12 02:53:02.441607 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.441548    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:53:02.535396 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.535363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5164d2-9413-460a-ab29-fc1521076c1a-openshift-service-ca\") pod \"ea5164d2-9413-460a-ab29-fc1521076c1a\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") "
Sep 12 02:53:02.535571 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.535446    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-maas-api-tls\") pod \"ea5164d2-9413-460a-ab29-fc1521076c1a\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") "
Sep 12 02:53:02.535571 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.535482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcdzm\" (UniqueName: \"kubernetes.io/projected/ea5164d2-9413-460a-ab29-fc1521076c1a-kube-api-access-wcdzm\") pod \"ea5164d2-9413-460a-ab29-fc1521076c1a\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") "
Sep 12 02:53:02.535571 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.535528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-metrics-certs\") pod \"ea5164d2-9413-460a-ab29-fc1521076c1a\" (UID: \"ea5164d2-9413-460a-ab29-fc1521076c1a\") "
Sep 12 02:53:02.535860 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.535835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5164d2-9413-460a-ab29-fc1521076c1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea5164d2-9413-460a-ab29-fc1521076c1a" (UID: "ea5164d2-9413-460a-ab29-fc1521076c1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:53:02.538716 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.538686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea5164d2-9413-460a-ab29-fc1521076c1a" (UID: "ea5164d2-9413-460a-ab29-fc1521076c1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:53:02.538716 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.538704    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea5164d2-9413-460a-ab29-fc1521076c1a" (UID: "ea5164d2-9413-460a-ab29-fc1521076c1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:53:02.538852 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.538718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5164d2-9413-460a-ab29-fc1521076c1a-kube-api-access-wcdzm" (OuterVolumeSpecName: "kube-api-access-wcdzm") pod "ea5164d2-9413-460a-ab29-fc1521076c1a" (UID: "ea5164d2-9413-460a-ab29-fc1521076c1a"). InnerVolumeSpecName "kube-api-access-wcdzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:53:02.636650 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.636614    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5164d2-9413-460a-ab29-fc1521076c1a-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:02.636650 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.636648    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:02.636650 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.636658    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcdzm\" (UniqueName: \"kubernetes.io/projected/ea5164d2-9413-460a-ab29-fc1521076c1a-kube-api-access-wcdzm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:02.636876 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.636667    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5164d2-9413-460a-ab29-fc1521076c1a-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:02.740743 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.740665    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea5164d2-9413-460a-ab29-fc1521076c1a" containerID="1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de" exitCode=0
Sep 12 02:53:02.740743 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.740709    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"
Sep 12 02:53:02.740904 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.740737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9" event={"ID":"ea5164d2-9413-460a-ab29-fc1521076c1a","Type":"ContainerDied","Data":"1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de"}
Sep 12 02:53:02.740904 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.740791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9" event={"ID":"ea5164d2-9413-460a-ab29-fc1521076c1a","Type":"ContainerDied","Data":"5cbd105ece35440d81d458d6f5e7c9b36e599418a30a6119179960da0d26fc0f"}
Sep 12 02:53:02.740904 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.740813    2579 scope.go:117] "RemoveContainer" containerID="1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de"
Sep 12 02:53:02.751052 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.751034    2579 scope.go:117] "RemoveContainer" containerID="1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de"
Sep 12 02:53:02.751366 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:53:02.751347    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de\": container with ID starting with 1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de not found: ID does not exist" containerID="1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de"
Sep 12 02:53:02.751463 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.751371    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de"} err="failed to get container status \"1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de\": rpc error: code = NotFound desc = could not find container \"1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de\": container with ID starting with 1c55f32f78c4b3ed907177a22e1118d6da4bc2e20c9ba8efb3aff8e2112963de not found: ID does not exist"
Sep 12 02:53:02.764174 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.764156    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"]
Sep 12 02:53:02.768074 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:02.768054    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-214c0e42-67f69b85f-9vlc9"]
Sep 12 02:53:03.249022 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:03.248992    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea5164d2-9413-460a-ab29-fc1521076c1a" path="/var/lib/kubelet/pods/ea5164d2-9413-460a-ab29-fc1521076c1a/volumes"
Sep 12 02:53:08.046953 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.046923    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"]
Sep 12 02:53:08.047354 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.047313    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea5164d2-9413-460a-ab29-fc1521076c1a" containerName="maas-api"
Sep 12 02:53:08.047354 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.047323    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea5164d2-9413-460a-ab29-fc1521076c1a" containerName="maas-api"
Sep 12 02:53:08.047426 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.047384    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea5164d2-9413-460a-ab29-fc1521076c1a" containerName="maas-api"
Sep 12 02:53:08.051730 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.051710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.053696 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.053678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-03cd9838-serving-cert\""
Sep 12 02:53:08.061137 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.061118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"]
Sep 12 02:53:08.179760 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.179730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-metrics-certs\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.179925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.179767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-openshift-service-ca\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.179925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.179796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-maas-api-tls\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.179925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.179838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxql\" (UniqueName: \"kubernetes.io/projected/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-kube-api-access-vbxql\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.280442 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.280407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-metrics-certs\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.280442 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.280440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-openshift-service-ca\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.280654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.280469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-maas-api-tls\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.280654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.280580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxql\" (UniqueName: \"kubernetes.io/projected/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-kube-api-access-vbxql\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.281284 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.281210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-openshift-service-ca\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.283008 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.282979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-maas-api-tls\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.283128 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.283014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-metrics-certs\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.288510 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.288490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxql\" (UniqueName: \"kubernetes.io/projected/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-kube-api-access-vbxql\") pod \"maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.363831 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.363772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:08.639865 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.639845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"]
Sep 12 02:53:08.642396 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:53:08.642370    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a9a33f_ff01_4b6f_ae77_e22f08129b97.slice/crio-b45984eccfbb1bd7779c5eadd37d736b448fd591794e5f1a8bff5c934595e339 WatchSource:0}: Error finding container b45984eccfbb1bd7779c5eadd37d736b448fd591794e5f1a8bff5c934595e339: Status 404 returned error can't find the container with id b45984eccfbb1bd7779c5eadd37d736b448fd591794e5f1a8bff5c934595e339
Sep 12 02:53:08.772544 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:08.772506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg" event={"ID":"f2a9a33f-ff01-4b6f-ae77-e22f08129b97","Type":"ContainerStarted","Data":"b45984eccfbb1bd7779c5eadd37d736b448fd591794e5f1a8bff5c934595e339"}
Sep 12 02:53:09.779965 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:09.779929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg" event={"ID":"f2a9a33f-ff01-4b6f-ae77-e22f08129b97","Type":"ContainerStarted","Data":"f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d"}
Sep 12 02:53:09.780363 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:09.780053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:09.796320 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:09.796264    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg" podStartSLOduration=1.433527013 podStartE2EDuration="1.796249626s" totalImagesPullingTime="362.722613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:53:09.794599603 +0000 UTC m=+1029.062633253" watchObservedRunningTime="2026-09-12 02:53:09.796249626 +0000 UTC m=+1029.064283469"
Sep 12 02:53:10.027354 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.027320    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"]
Sep 12 02:53:10.031028 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.030973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.033280 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.033248    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"openshift-service-ca.crt\""
Sep 12 02:53:10.033280 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.033270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"kube-root-ca.crt\""
Sep 12 02:53:10.033504 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.033279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"simulator-w0-b0573c-kserve-self-signed-certs\""
Sep 12 02:53:10.033680 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.033662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"default-dockercfg-lbnv8\""
Sep 12 02:53:10.040768 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.040745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"]
Sep 12 02:53:10.098802 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.098766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-model-cache\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.098976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.098815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-tmp-dir\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.098976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.098901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnq6n\" (UniqueName: \"kubernetes.io/projected/e89c3a97-7694-47be-818f-eeb429d25743-kube-api-access-wnq6n\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.098976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.098962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-home\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.099148 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.099019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-dshm\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.099148 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.099045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89c3a97-7694-47be-818f-eeb429d25743-tls-certs\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.200632 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.200598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-dshm\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.200632 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.200633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89c3a97-7694-47be-818f-eeb429d25743-tls-certs\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.200873 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.200697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-model-cache\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.200873 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.200733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-tmp-dir\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.200873 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.200801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnq6n\" (UniqueName: \"kubernetes.io/projected/e89c3a97-7694-47be-818f-eeb429d25743-kube-api-access-wnq6n\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.200873 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.200827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-home\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.201166 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.201141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-model-cache\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.201255 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.201155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-tmp-dir\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.201255 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.201203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-home\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.203566 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.203544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-dshm\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.203743 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.203727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89c3a97-7694-47be-818f-eeb429d25743-tls-certs\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.208730 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.208709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnq6n\" (UniqueName: \"kubernetes.io/projected/e89c3a97-7694-47be-818f-eeb429d25743-kube-api-access-wnq6n\") pod \"simulator-w0-b0573c-kserve-5b9d779465-j8rkf\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") " pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.340826 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.340746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:10.610997 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.610973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"]
Sep 12 02:53:10.614043 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:53:10.614019    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode89c3a97_7694_47be_818f_eeb429d25743.slice/crio-d049226e089d98a7328f823c00fb01616c7e628e79ddda713fd2cb66e9263b02 WatchSource:0}: Error finding container d049226e089d98a7328f823c00fb01616c7e628e79ddda713fd2cb66e9263b02: Status 404 returned error can't find the container with id d049226e089d98a7328f823c00fb01616c7e628e79ddda713fd2cb66e9263b02
Sep 12 02:53:10.787472 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.787438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf" event={"ID":"e89c3a97-7694-47be-818f-eeb429d25743","Type":"ContainerStarted","Data":"f68a866c8d54bd359b028779c5f132bbfcb5b0bba9c749de425f1af10371e2ab"}
Sep 12 02:53:10.787835 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.787482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf" event={"ID":"e89c3a97-7694-47be-818f-eeb429d25743","Type":"ContainerStarted","Data":"d049226e089d98a7328f823c00fb01616c7e628e79ddda713fd2cb66e9263b02"}
Sep 12 02:53:10.805999 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:10.805943    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf" podStartSLOduration=0.80592335 podStartE2EDuration="805.92335ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:53:10.804739413 +0000 UTC m=+1030.072773060" watchObservedRunningTime="2026-09-12 02:53:10.80592335 +0000 UTC m=+1030.073956998"
Sep 12 02:53:15.457498 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:15.457467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:53:15.795012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:15.794902    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:20.341103 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:20.341067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:20.341491 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:20.341136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:20.353651 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:20.353623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:20.849985 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:20.849955    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 02:53:38.008991 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.008959    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"]
Sep 12 02:53:38.009585 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.009223    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg" podUID="f2a9a33f-ff01-4b6f-ae77-e22f08129b97" containerName="maas-api" containerID="cri-o://f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d" gracePeriod=30
Sep 12 02:53:38.257570 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.257545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:38.360012 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.359917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-metrics-certs\") pod \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") "
Sep 12 02:53:38.360199 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.360033    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-maas-api-tls\") pod \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") "
Sep 12 02:53:38.360199 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.360063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-openshift-service-ca\") pod \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") "
Sep 12 02:53:38.360199 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.360129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbxql\" (UniqueName: \"kubernetes.io/projected/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-kube-api-access-vbxql\") pod \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\" (UID: \"f2a9a33f-ff01-4b6f-ae77-e22f08129b97\") "
Sep 12 02:53:38.360682 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.360650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2a9a33f-ff01-4b6f-ae77-e22f08129b97" (UID: "f2a9a33f-ff01-4b6f-ae77-e22f08129b97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:53:38.363829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.363802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2a9a33f-ff01-4b6f-ae77-e22f08129b97" (UID: "f2a9a33f-ff01-4b6f-ae77-e22f08129b97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:53:38.364060 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.364039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-kube-api-access-vbxql" (OuterVolumeSpecName: "kube-api-access-vbxql") pod "f2a9a33f-ff01-4b6f-ae77-e22f08129b97" (UID: "f2a9a33f-ff01-4b6f-ae77-e22f08129b97"). InnerVolumeSpecName "kube-api-access-vbxql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:53:38.364212 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.364049    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2a9a33f-ff01-4b6f-ae77-e22f08129b97" (UID: "f2a9a33f-ff01-4b6f-ae77-e22f08129b97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:53:38.461792 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.461757    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:38.461792 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.461789    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:38.462005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.461803    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbxql\" (UniqueName: \"kubernetes.io/projected/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-kube-api-access-vbxql\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:38.462005 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.461815    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2a9a33f-ff01-4b6f-ae77-e22f08129b97-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:53:38.928358 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.928325    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2a9a33f-ff01-4b6f-ae77-e22f08129b97" containerID="f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d" exitCode=0
Sep 12 02:53:38.928530 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.928366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg" event={"ID":"f2a9a33f-ff01-4b6f-ae77-e22f08129b97","Type":"ContainerDied","Data":"f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d"}
Sep 12 02:53:38.928530 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.928377    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"
Sep 12 02:53:38.928530 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.928441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg" event={"ID":"f2a9a33f-ff01-4b6f-ae77-e22f08129b97","Type":"ContainerDied","Data":"b45984eccfbb1bd7779c5eadd37d736b448fd591794e5f1a8bff5c934595e339"}
Sep 12 02:53:38.928530 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.928464    2579 scope.go:117] "RemoveContainer" containerID="f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d"
Sep 12 02:53:38.937797 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.937778    2579 scope.go:117] "RemoveContainer" containerID="f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d"
Sep 12 02:53:38.938067 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:53:38.938047    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d\": container with ID starting with f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d not found: ID does not exist" containerID="f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d"
Sep 12 02:53:38.938172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.938078    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d"} err="failed to get container status \"f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d\": rpc error: code = NotFound desc = could not find container \"f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d\": container with ID starting with f2ebb282e80e3aaa84e5177a713df79186f0a20b0a613acd1160c5f5be24eb1d not found: ID does not exist"
Sep 12 02:53:38.948211 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.948190    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"]
Sep 12 02:53:38.951870 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:38.951847    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03cd9838-755679bb4f-dr5bg"]
Sep 12 02:53:39.249740 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:39.249706    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a9a33f-ff01-4b6f-ae77-e22f08129b97" path="/var/lib/kubelet/pods/f2a9a33f-ff01-4b6f-ae77-e22f08129b97/volumes"
Sep 12 02:53:43.156281 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.156250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"]
Sep 12 02:53:43.156674 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.156620    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a9a33f-ff01-4b6f-ae77-e22f08129b97" containerName="maas-api"
Sep 12 02:53:43.156674 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.156631    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a9a33f-ff01-4b6f-ae77-e22f08129b97" containerName="maas-api"
Sep 12 02:53:43.156761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.156710    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a9a33f-ff01-4b6f-ae77-e22f08129b97" containerName="maas-api"
Sep 12 02:53:43.160889 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.160873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.163378 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.163360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1ac5b7fd-serving-cert\""
Sep 12 02:53:43.169034 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.169010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"]
Sep 12 02:53:43.306540 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.306507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-maas-api-tls\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.306697 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.306561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-metrics-certs\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.306697 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.306592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-openshift-service-ca\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.306697 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.306651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ptd\" (UniqueName: \"kubernetes.io/projected/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-kube-api-access-n6ptd\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.407980 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.407887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ptd\" (UniqueName: \"kubernetes.io/projected/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-kube-api-access-n6ptd\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.408178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.407977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-maas-api-tls\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.408178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.408057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-metrics-certs\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.408178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.408119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-openshift-service-ca\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.408872 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.408787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-openshift-service-ca\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.410657 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.410632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-maas-api-tls\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.410657 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.410646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-metrics-certs\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.415605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.415581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ptd\" (UniqueName: \"kubernetes.io/projected/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-kube-api-access-n6ptd\") pod \"maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.472563 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.472537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:43.752877 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.752856    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"]
Sep 12 02:53:43.754906 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:53:43.754878    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e510a6_a4c8_4ee7_b07a_b56d4181e1a9.slice/crio-1e6d5656baeeb48377ea635bd3e2b4587b566aba2d1a35eed59e1013a08b3d26 WatchSource:0}: Error finding container 1e6d5656baeeb48377ea635bd3e2b4587b566aba2d1a35eed59e1013a08b3d26: Status 404 returned error can't find the container with id 1e6d5656baeeb48377ea635bd3e2b4587b566aba2d1a35eed59e1013a08b3d26
Sep 12 02:53:43.957328 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:43.957286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz" event={"ID":"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9","Type":"ContainerStarted","Data":"1e6d5656baeeb48377ea635bd3e2b4587b566aba2d1a35eed59e1013a08b3d26"}
Sep 12 02:53:44.964623 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:44.964589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz" event={"ID":"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9","Type":"ContainerStarted","Data":"650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736"}
Sep 12 02:53:44.964990 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:44.964768    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:44.980721 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:44.980671    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz" podStartSLOduration=1.591942717 podStartE2EDuration="1.980657213s" totalImagesPullingTime="388.714496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:53:44.979505878 +0000 UTC m=+1064.247539526" watchObservedRunningTime="2026-09-12 02:53:44.980657213 +0000 UTC m=+1064.248690860"
Sep 12 02:53:45.341320 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:45.341233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:53:50.975226 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:50.975123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:53:58.510025 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.509947    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"]
Sep 12 02:53:58.513735 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.513716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.515880 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.515858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"simulator-w4-c6190b-kserve-self-signed-certs\""
Sep 12 02:53:58.516766 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.516746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"default-dockercfg-hl88b\""
Sep 12 02:53:58.516836 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.516783    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"kube-root-ca.crt\""
Sep 12 02:53:58.516882 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.516756    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"openshift-service-ca.crt\""
Sep 12 02:53:58.522172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.522153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"]
Sep 12 02:53:58.546598 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.546580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-model-cache\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.546686 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.546609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tls-certs\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.546686 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.546630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-home\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.546686 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.546647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tmp-dir\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.546790 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.546729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlsr\" (UniqueName: \"kubernetes.io/projected/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-kube-api-access-9mlsr\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.546832 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.546817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-dshm\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.648607 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.648583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-dshm\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.648762 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.648626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-model-cache\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.648762 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.648650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tls-certs\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.648762 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.648670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-home\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.648762 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.648688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tmp-dir\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.648762 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.648736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlsr\" (UniqueName: \"kubernetes.io/projected/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-kube-api-access-9mlsr\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.649121 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.649082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-home\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.649210 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.649080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-model-cache\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.649210 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.649161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tmp-dir\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.650970 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.650951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-dshm\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.651151 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.651132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tls-certs\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.656475 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.656455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlsr\" (UniqueName: \"kubernetes.io/projected/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-kube-api-access-9mlsr\") pod \"simulator-w4-c6190b-kserve-545658cb98-qscr7\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") " pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:58.824400 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:58.824322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:53:59.095272 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:53:59.095244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"]
Sep 12 02:53:59.097350 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:53:59.097320    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0217ee9c_e6ad_4b45_b748_3fda3e9cc791.slice/crio-9f70a1fadd4c4aec46ae98507bfee647fbc19dcb81d6cdba2a0bd3a1c4459f04 WatchSource:0}: Error finding container 9f70a1fadd4c4aec46ae98507bfee647fbc19dcb81d6cdba2a0bd3a1c4459f04: Status 404 returned error can't find the container with id 9f70a1fadd4c4aec46ae98507bfee647fbc19dcb81d6cdba2a0bd3a1c4459f04
Sep 12 02:54:00.045958 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:00.045925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7" event={"ID":"0217ee9c-e6ad-4b45-b748-3fda3e9cc791","Type":"ContainerStarted","Data":"b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700"}
Sep 12 02:54:00.046456 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:00.045970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7" event={"ID":"0217ee9c-e6ad-4b45-b748-3fda3e9cc791","Type":"ContainerStarted","Data":"9f70a1fadd4c4aec46ae98507bfee647fbc19dcb81d6cdba2a0bd3a1c4459f04"}
Sep 12 02:54:00.063181 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:00.063138    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7" podStartSLOduration=2.063122336 podStartE2EDuration="2.063122336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:54:00.060665231 +0000 UTC m=+1079.328698880" watchObservedRunningTime="2026-09-12 02:54:00.063122336 +0000 UTC m=+1079.331155982"
Sep 12 02:54:02.952491 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:02.952453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:54:08.825447 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:08.825397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:54:08.825947 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:08.825461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:54:08.838197 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:08.838173    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:54:09.101995 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:09.101912    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 02:54:13.565268 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.565237    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"]
Sep 12 02:54:13.565737 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.565438    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz" podUID="73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" containerName="maas-api" containerID="cri-o://650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736" gracePeriod=30
Sep 12 02:54:13.812520 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.812496    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:54:13.890469 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.890373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6ptd\" (UniqueName: \"kubernetes.io/projected/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-kube-api-access-n6ptd\") pod \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") "
Sep 12 02:54:13.890469 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.890448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-metrics-certs\") pod \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") "
Sep 12 02:54:13.890707 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.890523    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-openshift-service-ca\") pod \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") "
Sep 12 02:54:13.890707 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.890574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-maas-api-tls\") pod \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\" (UID: \"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9\") "
Sep 12 02:54:13.890903 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.890871    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" (UID: "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:54:13.892771 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.892743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" (UID: "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:54:13.893125 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.893106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" (UID: "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:54:13.893219 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.893146    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-kube-api-access-n6ptd" (OuterVolumeSpecName: "kube-api-access-n6ptd") pod "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" (UID: "73e510a6-a4c8-4ee7-b07a-b56d4181e1a9"). InnerVolumeSpecName "kube-api-access-n6ptd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:54:13.991850 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.991815    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:54:13.991850 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.991851    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ptd\" (UniqueName: \"kubernetes.io/projected/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-kube-api-access-n6ptd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:54:13.992060 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.991861    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:54:13.992060 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:13.991870    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:54:14.117075 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.117037    2579 generic.go:356] "Generic (PLEG): container finished" podID="73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" containerID="650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736" exitCode=0
Sep 12 02:54:14.117304 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.117088    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"
Sep 12 02:54:14.117304 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.117130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz" event={"ID":"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9","Type":"ContainerDied","Data":"650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736"}
Sep 12 02:54:14.117304 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.117200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz" event={"ID":"73e510a6-a4c8-4ee7-b07a-b56d4181e1a9","Type":"ContainerDied","Data":"1e6d5656baeeb48377ea635bd3e2b4587b566aba2d1a35eed59e1013a08b3d26"}
Sep 12 02:54:14.117304 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.117230    2579 scope.go:117] "RemoveContainer" containerID="650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736"
Sep 12 02:54:14.127588 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.127571    2579 scope.go:117] "RemoveContainer" containerID="650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736"
Sep 12 02:54:14.127891 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:54:14.127864    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736\": container with ID starting with 650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736 not found: ID does not exist" containerID="650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736"
Sep 12 02:54:14.127998 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.127901    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736"} err="failed to get container status \"650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736\": rpc error: code = NotFound desc = could not find container \"650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736\": container with ID starting with 650d00ea840f51ba3bc3318e70ae470774ad99901eebed3830e78d5979881736 not found: ID does not exist"
Sep 12 02:54:14.138106 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.138074    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"]
Sep 12 02:54:14.145081 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:14.145034    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ac5b7fd-66997c685c-j6bqz"]
Sep 12 02:54:15.249385 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:15.249346    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" path="/var/lib/kubelet/pods/73e510a6-a4c8-4ee7-b07a-b56d4181e1a9/volumes"
Sep 12 02:54:21.044925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:21.044895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:54:22.243778 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.243746    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"]
Sep 12 02:54:22.244351 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.244328    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" containerName="maas-api"
Sep 12 02:54:22.244515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.244503    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" containerName="maas-api"
Sep 12 02:54:22.244712 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.244701    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e510a6-a4c8-4ee7-b07a-b56d4181e1a9" containerName="maas-api"
Sep 12 02:54:22.250440 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.250420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.252815 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.252798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bc3f9c2f-serving-cert\""
Sep 12 02:54:22.259572 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.259438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"]
Sep 12 02:54:22.368534 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.368503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2lr\" (UniqueName: \"kubernetes.io/projected/b99e04ea-4c97-434a-b903-b101be06d34e-kube-api-access-2h2lr\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.368726 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.368620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-metrics-certs\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.368726 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.368651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99e04ea-4c97-434a-b903-b101be06d34e-openshift-service-ca\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.368726 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.368681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.469793 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.469762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-metrics-certs\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.469793 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.469796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99e04ea-4c97-434a-b903-b101be06d34e-openshift-service-ca\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.469976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.469826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.469976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.469894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2lr\" (UniqueName: \"kubernetes.io/projected/b99e04ea-4c97-434a-b903-b101be06d34e-kube-api-access-2h2lr\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.470057 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:54:22.470006    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-serving-cert: secret "maas-api-e2e-derive-bc3f9c2f-serving-cert" not found
Sep 12 02:54:22.470127 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:54:22.470106    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls podName:b99e04ea-4c97-434a-b903-b101be06d34e nodeName:}" failed. No retries permitted until 2026-09-12 02:54:22.970069664 +0000 UTC m=+1102.238103293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls") pod "maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9" (UID: "b99e04ea-4c97-434a-b903-b101be06d34e") : secret "maas-api-e2e-derive-bc3f9c2f-serving-cert" not found
Sep 12 02:54:22.470515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.470491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99e04ea-4c97-434a-b903-b101be06d34e-openshift-service-ca\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.472519 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.472491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-metrics-certs\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.480109 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.480072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2lr\" (UniqueName: \"kubernetes.io/projected/b99e04ea-4c97-434a-b903-b101be06d34e-kube-api-access-2h2lr\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.975940 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.975908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:22.978646 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:22.978616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls\") pod \"maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:23.165425 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:23.165396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:23.437512 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:23.437486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"]
Sep 12 02:54:23.439209 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:54:23.439183    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99e04ea_4c97_434a_b903_b101be06d34e.slice/crio-cf2c81ef4dfec952b4d6b4dc5b0d53952bcd0e3f2d3dc22cd670ae145d3a7c89 WatchSource:0}: Error finding container cf2c81ef4dfec952b4d6b4dc5b0d53952bcd0e3f2d3dc22cd670ae145d3a7c89: Status 404 returned error can't find the container with id cf2c81ef4dfec952b4d6b4dc5b0d53952bcd0e3f2d3dc22cd670ae145d3a7c89
Sep 12 02:54:24.171839 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:24.171759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9" event={"ID":"b99e04ea-4c97-434a-b903-b101be06d34e","Type":"ContainerStarted","Data":"b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d"}
Sep 12 02:54:24.171839 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:24.171808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9" event={"ID":"b99e04ea-4c97-434a-b903-b101be06d34e","Type":"ContainerStarted","Data":"cf2c81ef4dfec952b4d6b4dc5b0d53952bcd0e3f2d3dc22cd670ae145d3a7c89"}
Sep 12 02:54:24.172029 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:24.171891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:24.188794 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:24.188733    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9" podStartSLOduration=1.821943143 podStartE2EDuration="2.188717515s" totalImagesPullingTime="366.774372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:54:24.187191473 +0000 UTC m=+1103.455225122" watchObservedRunningTime="2026-09-12 02:54:24.188717515 +0000 UTC m=+1103.456751161"
Sep 12 02:54:27.398752 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.398721    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"]
Sep 12 02:54:27.402312 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.402295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.405431 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.405402    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4b3be7\"/\"openshift-service-ca.crt\""
Sep 12 02:54:27.405431 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.405425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4b3be7\"/\"default-dockercfg-nn5v6\""
Sep 12 02:54:27.405619 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.405403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4b3be7\"/\"autores-model-4b3be7-kserve-self-signed-certs\""
Sep 12 02:54:27.405619 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.405458    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4b3be7\"/\"kube-root-ca.crt\""
Sep 12 02:54:27.411889 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.411862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"]
Sep 12 02:54:27.519990 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.519957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-model-cache\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.520196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.520002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6pw4\" (UniqueName: \"kubernetes.io/projected/a525478b-e010-4982-ac65-f440b202e892-kube-api-access-d6pw4\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.520196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.520032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-home\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.520196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.520073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-dshm\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.520196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.520146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a525478b-e010-4982-ac65-f440b202e892-tls-certs\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.520196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.520169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-tmp-dir\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.620899 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.620864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6pw4\" (UniqueName: \"kubernetes.io/projected/a525478b-e010-4982-ac65-f440b202e892-kube-api-access-d6pw4\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.620899 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.620901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-home\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.621172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.620926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-dshm\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.621172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.620981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a525478b-e010-4982-ac65-f440b202e892-tls-certs\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.621172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.621016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-tmp-dir\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.621172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.621084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-model-cache\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.621405 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.621384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-home\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.621465 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.621453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-tmp-dir\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.621511 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.621492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-model-cache\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.623483 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.623461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-dshm\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.623690 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.623671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a525478b-e010-4982-ac65-f440b202e892-tls-certs\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.628845 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.628826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6pw4\" (UniqueName: \"kubernetes.io/projected/a525478b-e010-4982-ac65-f440b202e892-kube-api-access-d6pw4\") pod \"autores-model-4b3be7-kserve-59995cd9f5-mnpb8\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") " pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.712737 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.712712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:27.984115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:27.984044    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"]
Sep 12 02:54:27.986448 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:54:27.986417    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda525478b_e010_4982_ac65_f440b202e892.slice/crio-7e5216bdc58c0611c41f91b332ed396dac7c421d85bfca5d8a805071f8afb5dd WatchSource:0}: Error finding container 7e5216bdc58c0611c41f91b332ed396dac7c421d85bfca5d8a805071f8afb5dd: Status 404 returned error can't find the container with id 7e5216bdc58c0611c41f91b332ed396dac7c421d85bfca5d8a805071f8afb5dd
Sep 12 02:54:28.195171 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:28.195138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8" event={"ID":"a525478b-e010-4982-ac65-f440b202e892","Type":"ContainerStarted","Data":"807a6f3ac4d001ae9a5b0c4ed6f73c2195ff471df15ee88ab6c97765904171ec"}
Sep 12 02:54:28.195341 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:28.195184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8" event={"ID":"a525478b-e010-4982-ac65-f440b202e892","Type":"ContainerStarted","Data":"7e5216bdc58c0611c41f91b332ed396dac7c421d85bfca5d8a805071f8afb5dd"}
Sep 12 02:54:28.211957 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:28.211911    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8" podStartSLOduration=1.211896141 podStartE2EDuration="1.211896141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:54:28.210001218 +0000 UTC m=+1107.478034865" watchObservedRunningTime="2026-09-12 02:54:28.211896141 +0000 UTC m=+1107.479929787"
Sep 12 02:54:30.183103 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:30.183069    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:54:37.713817 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:37.713777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:37.714242 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:37.713832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:37.726523 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:37.726503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:38.260049 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:38.260017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:54:41.713359 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.713329    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"]
Sep 12 02:54:41.717508 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.717484    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.720153 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.720123    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a07b10\"/\"openshift-service-ca.crt\""
Sep 12 02:54:41.720299 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.720131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a07b10\"/\"kube-root-ca.crt\""
Sep 12 02:54:41.720988 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.720968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a07b10\"/\"default-dockercfg-qt4ls\""
Sep 12 02:54:41.721076 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.721020    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a07b10\"/\"test-model-a07b10-kserve-self-signed-certs\""
Sep 12 02:54:41.724968 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.724949    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"]
Sep 12 02:54:41.753470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.753442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-home\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.753624 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.753489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tmp-dir\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.753624 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.753520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-dshm\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.753624 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.753562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczc4\" (UniqueName: \"kubernetes.io/projected/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-kube-api-access-sczc4\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.753624 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.753590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tls-certs\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.753840 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.753741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-model-cache\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855249 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-model-cache\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855428 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-home\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855428 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tmp-dir\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855428 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-dshm\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855428 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczc4\" (UniqueName: \"kubernetes.io/projected/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-kube-api-access-sczc4\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855428 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tls-certs\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855690 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-model-cache\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855738 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tmp-dir\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.855784 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.855762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-home\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.857698 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.857673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-dshm\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.857851 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.857834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tls-certs\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:41.862760 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:41.862736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczc4\" (UniqueName: \"kubernetes.io/projected/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-kube-api-access-sczc4\") pod \"test-model-a07b10-kserve-7f4bf5d7d7-65x7n\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") " pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:42.028798 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:42.028713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:42.299854 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:42.299648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"]
Sep 12 02:54:42.302679 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:54:42.302640    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b619077_8fa2_4c8f_9115_cd0c5785a4d3.slice/crio-6384ea6e749d80ca943fe88ac999f1d32dc854171822311378be40c90095350f WatchSource:0}: Error finding container 6384ea6e749d80ca943fe88ac999f1d32dc854171822311378be40c90095350f: Status 404 returned error can't find the container with id 6384ea6e749d80ca943fe88ac999f1d32dc854171822311378be40c90095350f
Sep 12 02:54:43.274403 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:43.274364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n" event={"ID":"5b619077-8fa2-4c8f-9115-cd0c5785a4d3","Type":"ContainerStarted","Data":"8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53"}
Sep 12 02:54:43.274403 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:43.274404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n" event={"ID":"5b619077-8fa2-4c8f-9115-cd0c5785a4d3","Type":"ContainerStarted","Data":"6384ea6e749d80ca943fe88ac999f1d32dc854171822311378be40c90095350f"}
Sep 12 02:54:43.294038 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:43.293984    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n" podStartSLOduration=2.293964997 podStartE2EDuration="2.293964997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:54:43.2914486 +0000 UTC m=+1122.559482247" watchObservedRunningTime="2026-09-12 02:54:43.293964997 +0000 UTC m=+1122.561998645"
Sep 12 02:54:43.644681 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:43.644604    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:54:52.028961 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:52.028922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:52.029446 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:52.028978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:52.041564 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:52.041542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:54:52.330907 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:54:52.330811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:55:00.746301 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:00.746269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:55:04.703610 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:04.703584    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"]
Sep 12 02:55:04.704002 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:04.703766    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9" podUID="b99e04ea-4c97-434a-b903-b101be06d34e" containerName="maas-api" containerID="cri-o://b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d" gracePeriod=30
Sep 12 02:55:04.948468 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:04.948445    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:55:05.079400 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.079317    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99e04ea-4c97-434a-b903-b101be06d34e-openshift-service-ca\") pod \"b99e04ea-4c97-434a-b903-b101be06d34e\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") "
Sep 12 02:55:05.079536 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.079464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls\") pod \"b99e04ea-4c97-434a-b903-b101be06d34e\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") "
Sep 12 02:55:05.079536 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.079521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-metrics-certs\") pod \"b99e04ea-4c97-434a-b903-b101be06d34e\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") "
Sep 12 02:55:05.079605 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.079569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h2lr\" (UniqueName: \"kubernetes.io/projected/b99e04ea-4c97-434a-b903-b101be06d34e-kube-api-access-2h2lr\") pod \"b99e04ea-4c97-434a-b903-b101be06d34e\" (UID: \"b99e04ea-4c97-434a-b903-b101be06d34e\") "
Sep 12 02:55:05.079685 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.079666    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99e04ea-4c97-434a-b903-b101be06d34e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b99e04ea-4c97-434a-b903-b101be06d34e" (UID: "b99e04ea-4c97-434a-b903-b101be06d34e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:55:05.079874 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.079859    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99e04ea-4c97-434a-b903-b101be06d34e-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:05.081746 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.081718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b99e04ea-4c97-434a-b903-b101be06d34e" (UID: "b99e04ea-4c97-434a-b903-b101be06d34e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:55:05.081875 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.081765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b99e04ea-4c97-434a-b903-b101be06d34e" (UID: "b99e04ea-4c97-434a-b903-b101be06d34e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:55:05.081875 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.081773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99e04ea-4c97-434a-b903-b101be06d34e-kube-api-access-2h2lr" (OuterVolumeSpecName: "kube-api-access-2h2lr") pod "b99e04ea-4c97-434a-b903-b101be06d34e" (UID: "b99e04ea-4c97-434a-b903-b101be06d34e"). InnerVolumeSpecName "kube-api-access-2h2lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:55:05.181248 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.181219    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:05.181248 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.181245    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2lr\" (UniqueName: \"kubernetes.io/projected/b99e04ea-4c97-434a-b903-b101be06d34e-kube-api-access-2h2lr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:05.181248 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.181256    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99e04ea-4c97-434a-b903-b101be06d34e-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:05.387369 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.387296    2579 generic.go:356] "Generic (PLEG): container finished" podID="b99e04ea-4c97-434a-b903-b101be06d34e" containerID="b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d" exitCode=0
Sep 12 02:55:05.387531 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.387394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9" event={"ID":"b99e04ea-4c97-434a-b903-b101be06d34e","Type":"ContainerDied","Data":"b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d"}
Sep 12 02:55:05.387531 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.387406    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"
Sep 12 02:55:05.387531 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.387435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9" event={"ID":"b99e04ea-4c97-434a-b903-b101be06d34e","Type":"ContainerDied","Data":"cf2c81ef4dfec952b4d6b4dc5b0d53952bcd0e3f2d3dc22cd670ae145d3a7c89"}
Sep 12 02:55:05.387531 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.387456    2579 scope.go:117] "RemoveContainer" containerID="b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d"
Sep 12 02:55:05.396789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.396752    2579 scope.go:117] "RemoveContainer" containerID="b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d"
Sep 12 02:55:05.397063 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:55:05.397044    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d\": container with ID starting with b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d not found: ID does not exist" containerID="b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d"
Sep 12 02:55:05.397149 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.397073    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d"} err="failed to get container status \"b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d\": rpc error: code = NotFound desc = could not find container \"b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d\": container with ID starting with b0285631b31b8d2093919bfe24486224f32fd6e3e176c9aa4d1c7aa349df824d not found: ID does not exist"
Sep 12 02:55:05.404288 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.404268    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"]
Sep 12 02:55:05.411658 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:05.411639    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc3f9c2f-6d77844c65-6zxv9"]
Sep 12 02:55:07.249698 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:07.249667    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b99e04ea-4c97-434a-b903-b101be06d34e" path="/var/lib/kubelet/pods/b99e04ea-4c97-434a-b903-b101be06d34e/volumes"
Sep 12 02:55:10.755321 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:10.755286    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"]
Sep 12 02:55:10.755775 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:10.755508    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z" podUID="98a1a921-9247-4e57-b27b-b4468b03ce42" containerName="maas-api" containerID="cri-o://4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0" gracePeriod=30
Sep 12 02:55:11.002926 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.002897    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:55:11.032484 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.032418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcrnk\" (UniqueName: \"kubernetes.io/projected/98a1a921-9247-4e57-b27b-b4468b03ce42-kube-api-access-vcrnk\") pod \"98a1a921-9247-4e57-b27b-b4468b03ce42\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") "
Sep 12 02:55:11.032616 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.032500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls\") pod \"98a1a921-9247-4e57-b27b-b4468b03ce42\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") "
Sep 12 02:55:11.032684 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.032615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1a921-9247-4e57-b27b-b4468b03ce42-openshift-service-ca\") pod \"98a1a921-9247-4e57-b27b-b4468b03ce42\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") "
Sep 12 02:55:11.032684 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.032650    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-metrics-certs\") pod \"98a1a921-9247-4e57-b27b-b4468b03ce42\" (UID: \"98a1a921-9247-4e57-b27b-b4468b03ce42\") "
Sep 12 02:55:11.032976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.032950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a1a921-9247-4e57-b27b-b4468b03ce42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98a1a921-9247-4e57-b27b-b4468b03ce42" (UID: "98a1a921-9247-4e57-b27b-b4468b03ce42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:55:11.034560 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.034539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a1a921-9247-4e57-b27b-b4468b03ce42-kube-api-access-vcrnk" (OuterVolumeSpecName: "kube-api-access-vcrnk") pod "98a1a921-9247-4e57-b27b-b4468b03ce42" (UID: "98a1a921-9247-4e57-b27b-b4468b03ce42"). InnerVolumeSpecName "kube-api-access-vcrnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:55:11.035262 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.035238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98a1a921-9247-4e57-b27b-b4468b03ce42" (UID: "98a1a921-9247-4e57-b27b-b4468b03ce42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:55:11.035262 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.035252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98a1a921-9247-4e57-b27b-b4468b03ce42" (UID: "98a1a921-9247-4e57-b27b-b4468b03ce42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:55:11.134184 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.134155    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcrnk\" (UniqueName: \"kubernetes.io/projected/98a1a921-9247-4e57-b27b-b4468b03ce42-kube-api-access-vcrnk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:11.134184 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.134182    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:11.134369 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.134193    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a1a921-9247-4e57-b27b-b4468b03ce42-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:11.134369 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.134202    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a1a921-9247-4e57-b27b-b4468b03ce42-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:11.422415 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.422330    2579 generic.go:356] "Generic (PLEG): container finished" podID="98a1a921-9247-4e57-b27b-b4468b03ce42" containerID="4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0" exitCode=0
Sep 12 02:55:11.422415 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.422375    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"
Sep 12 02:55:11.422601 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.422425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z" event={"ID":"98a1a921-9247-4e57-b27b-b4468b03ce42","Type":"ContainerDied","Data":"4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0"}
Sep 12 02:55:11.422601 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.422474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z" event={"ID":"98a1a921-9247-4e57-b27b-b4468b03ce42","Type":"ContainerDied","Data":"bcb39aa91037b950c6eb073abae5d94f5aaf76d8f25f86802aafb72a5c4b5167"}
Sep 12 02:55:11.422601 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.422500    2579 scope.go:117] "RemoveContainer" containerID="4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0"
Sep 12 02:55:11.431852 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.431835    2579 scope.go:117] "RemoveContainer" containerID="4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0"
Sep 12 02:55:11.432141 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:55:11.432124    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0\": container with ID starting with 4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0 not found: ID does not exist" containerID="4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0"
Sep 12 02:55:11.432188 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.432151    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0"} err="failed to get container status \"4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0\": rpc error: code = NotFound desc = could not find container \"4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0\": container with ID starting with 4dbdfb5998073fcca3696f5b2019c18da04730e05dc9244acfdd07f4520233a0 not found: ID does not exist"
Sep 12 02:55:11.439648 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.439627    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"]
Sep 12 02:55:11.447949 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:11.447929    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4b3be7-7b7944c94d-8lm5z"]
Sep 12 02:55:13.249347 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:13.249318    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a1a921-9247-4e57-b27b-b4468b03ce42" path="/var/lib/kubelet/pods/98a1a921-9247-4e57-b27b-b4468b03ce42/volumes"
Sep 12 02:55:16.362777 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:16.362741    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"]
Sep 12 02:55:16.948712 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:16.948679    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:55:31.382994 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.382883    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8" podUID="a525478b-e010-4982-ac65-f440b202e892" containerName="main" containerID="cri-o://807a6f3ac4d001ae9a5b0c4ed6f73c2195ff471df15ee88ab6c97765904171ec" gracePeriod=45
Sep 12 02:55:31.526319 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.526288    2579 generic.go:356] "Generic (PLEG): container finished" podID="a525478b-e010-4982-ac65-f440b202e892" containerID="807a6f3ac4d001ae9a5b0c4ed6f73c2195ff471df15ee88ab6c97765904171ec" exitCode=0
Sep 12 02:55:31.526503 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.526341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8" event={"ID":"a525478b-e010-4982-ac65-f440b202e892","Type":"ContainerDied","Data":"807a6f3ac4d001ae9a5b0c4ed6f73c2195ff471df15ee88ab6c97765904171ec"}
Sep 12 02:55:31.644751 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.644694    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:55:31.721710 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.721677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-home\") pod \"a525478b-e010-4982-ac65-f440b202e892\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") "
Sep 12 02:55:31.721877 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.721731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6pw4\" (UniqueName: \"kubernetes.io/projected/a525478b-e010-4982-ac65-f440b202e892-kube-api-access-d6pw4\") pod \"a525478b-e010-4982-ac65-f440b202e892\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") "
Sep 12 02:55:31.721877 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.721769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-model-cache\") pod \"a525478b-e010-4982-ac65-f440b202e892\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") "
Sep 12 02:55:31.721877 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.721819    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a525478b-e010-4982-ac65-f440b202e892-tls-certs\") pod \"a525478b-e010-4982-ac65-f440b202e892\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") "
Sep 12 02:55:31.722053 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.721880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-tmp-dir\") pod \"a525478b-e010-4982-ac65-f440b202e892\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") "
Sep 12 02:55:31.722053 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.721907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-dshm\") pod \"a525478b-e010-4982-ac65-f440b202e892\" (UID: \"a525478b-e010-4982-ac65-f440b202e892\") "
Sep 12 02:55:31.722053 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.721927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-home" (OuterVolumeSpecName: "home") pod "a525478b-e010-4982-ac65-f440b202e892" (UID: "a525478b-e010-4982-ac65-f440b202e892"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:31.722053 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.722038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-model-cache" (OuterVolumeSpecName: "model-cache") pod "a525478b-e010-4982-ac65-f440b202e892" (UID: "a525478b-e010-4982-ac65-f440b202e892"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:31.722321 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.722154    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a525478b-e010-4982-ac65-f440b202e892" (UID: "a525478b-e010-4982-ac65-f440b202e892"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:31.722362 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.722341    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:31.722411 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.722360    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:31.722411 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.722373    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:31.724207 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.724188    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-dshm" (OuterVolumeSpecName: "dshm") pod "a525478b-e010-4982-ac65-f440b202e892" (UID: "a525478b-e010-4982-ac65-f440b202e892"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:31.724291 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.724226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a525478b-e010-4982-ac65-f440b202e892-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a525478b-e010-4982-ac65-f440b202e892" (UID: "a525478b-e010-4982-ac65-f440b202e892"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:55:31.724291 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.724276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a525478b-e010-4982-ac65-f440b202e892-kube-api-access-d6pw4" (OuterVolumeSpecName: "kube-api-access-d6pw4") pod "a525478b-e010-4982-ac65-f440b202e892" (UID: "a525478b-e010-4982-ac65-f440b202e892"). InnerVolumeSpecName "kube-api-access-d6pw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:55:31.822845 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.822818    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6pw4\" (UniqueName: \"kubernetes.io/projected/a525478b-e010-4982-ac65-f440b202e892-kube-api-access-d6pw4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:31.822845 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.822843    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a525478b-e010-4982-ac65-f440b202e892-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:31.822845 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:31.822854    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a525478b-e010-4982-ac65-f440b202e892-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:55:32.532589 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:32.532560    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"
Sep 12 02:55:32.533020 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:32.532567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8" event={"ID":"a525478b-e010-4982-ac65-f440b202e892","Type":"ContainerDied","Data":"7e5216bdc58c0611c41f91b332ed396dac7c421d85bfca5d8a805071f8afb5dd"}
Sep 12 02:55:32.533020 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:32.532695    2579 scope.go:117] "RemoveContainer" containerID="807a6f3ac4d001ae9a5b0c4ed6f73c2195ff471df15ee88ab6c97765904171ec"
Sep 12 02:55:32.554043 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:32.554019    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"]
Sep 12 02:55:32.559501 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:32.559478    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4b3be7/autores-model-4b3be7-kserve-59995cd9f5-mnpb8"]
Sep 12 02:55:33.249675 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:33.249639    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a525478b-e010-4982-ac65-f440b202e892" path="/var/lib/kubelet/pods/a525478b-e010-4982-ac65-f440b202e892/volumes"
Sep 12 02:55:44.411574 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.411546    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"]
Sep 12 02:55:44.412300 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412275    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a1a921-9247-4e57-b27b-b4468b03ce42" containerName="maas-api"
Sep 12 02:55:44.412431 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412302    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a1a921-9247-4e57-b27b-b4468b03ce42" containerName="maas-api"
Sep 12 02:55:44.412431 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412317    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a525478b-e010-4982-ac65-f440b202e892" containerName="main"
Sep 12 02:55:44.412431 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412325    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a525478b-e010-4982-ac65-f440b202e892" containerName="main"
Sep 12 02:55:44.412431 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412343    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99e04ea-4c97-434a-b903-b101be06d34e" containerName="maas-api"
Sep 12 02:55:44.412431 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412351    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99e04ea-4c97-434a-b903-b101be06d34e" containerName="maas-api"
Sep 12 02:55:44.412704 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412481    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a1a921-9247-4e57-b27b-b4468b03ce42" containerName="maas-api"
Sep 12 02:55:44.412704 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412495    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a525478b-e010-4982-ac65-f440b202e892" containerName="main"
Sep 12 02:55:44.412704 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.412507    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99e04ea-4c97-434a-b903-b101be06d34e" containerName="maas-api"
Sep 12 02:55:44.416924 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.416902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.420568 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.420548    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-450f2a-serving-cert\""
Sep 12 02:55:44.428811 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.428786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"]
Sep 12 02:55:44.538426 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.538386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9w9f\" (UniqueName: \"kubernetes.io/projected/55c25822-2f10-4212-b2c5-e6b630607e6d-kube-api-access-f9w9f\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.538582 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.538501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-maas-api-tls\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.538582 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.538538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c25822-2f10-4212-b2c5-e6b630607e6d-openshift-service-ca\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.538582 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.538581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-metrics-certs\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.639016 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.638985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-maas-api-tls\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.639203 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.639025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c25822-2f10-4212-b2c5-e6b630607e6d-openshift-service-ca\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.639203 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.639058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-metrics-certs\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.639203 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.639086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9w9f\" (UniqueName: \"kubernetes.io/projected/55c25822-2f10-4212-b2c5-e6b630607e6d-kube-api-access-f9w9f\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.639757 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.639726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c25822-2f10-4212-b2c5-e6b630607e6d-openshift-service-ca\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.641612 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.641585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-maas-api-tls\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.641738 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.641694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-metrics-certs\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.646980 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.646954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9w9f\" (UniqueName: \"kubernetes.io/projected/55c25822-2f10-4212-b2c5-e6b630607e6d-kube-api-access-f9w9f\") pod \"maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:44.729217 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:44.729180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:45.003290 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:45.003113    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"]
Sep 12 02:55:45.005587 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:55:45.005549    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c25822_2f10_4212_b2c5_e6b630607e6d.slice/crio-6fd93177ae6616317b80f3b98bc7e1bff1339770dc5dceb3be53a14de83d810f WatchSource:0}: Error finding container 6fd93177ae6616317b80f3b98bc7e1bff1339770dc5dceb3be53a14de83d810f: Status 404 returned error can't find the container with id 6fd93177ae6616317b80f3b98bc7e1bff1339770dc5dceb3be53a14de83d810f
Sep 12 02:55:45.606909 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:45.606864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl" event={"ID":"55c25822-2f10-4212-b2c5-e6b630607e6d","Type":"ContainerStarted","Data":"b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4"}
Sep 12 02:55:45.607395 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:45.606922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl" event={"ID":"55c25822-2f10-4212-b2c5-e6b630607e6d","Type":"ContainerStarted","Data":"6fd93177ae6616317b80f3b98bc7e1bff1339770dc5dceb3be53a14de83d810f"}
Sep 12 02:55:45.607395 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:45.607011    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:45.623848 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:45.623793    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl" podStartSLOduration=1.246226365 podStartE2EDuration="1.62377601s" totalImagesPullingTime="377.549645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:55:45.623064229 +0000 UTC m=+1184.891097877" watchObservedRunningTime="2026-09-12 02:55:45.62377601 +0000 UTC m=+1184.891809658"
Sep 12 02:55:51.616679 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:51.616648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:55:54.250502 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.250466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:55:54.323692 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.323661    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"]
Sep 12 02:55:54.327330 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.327311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.329624 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.329604    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"premium-w0-b0573c-kserve-self-signed-certs\""
Sep 12 02:55:54.338485 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.338466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"]
Sep 12 02:55:54.438656 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.438630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-tmp-dir\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.438824 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.438666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbm6\" (UniqueName: \"kubernetes.io/projected/b7a71477-fd5b-47ed-9949-a30143d02142-kube-api-access-tdbm6\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.438824 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.438689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a71477-fd5b-47ed-9949-a30143d02142-tls-certs\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.438824 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.438770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-home\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.438943 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.438833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-model-cache\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.438943 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.438867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-dshm\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540074 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.539996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-home\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540074 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-model-cache\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540074 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-dshm\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540339 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-tmp-dir\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540339 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbm6\" (UniqueName: \"kubernetes.io/projected/b7a71477-fd5b-47ed-9949-a30143d02142-kube-api-access-tdbm6\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540339 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a71477-fd5b-47ed-9949-a30143d02142-tls-certs\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540508 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-model-cache\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540508 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-tmp-dir\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.540620 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.540557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-home\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.542636 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.542607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-dshm\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.542781 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.542764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a71477-fd5b-47ed-9949-a30143d02142-tls-certs\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.551527 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.551503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbm6\" (UniqueName: \"kubernetes.io/projected/b7a71477-fd5b-47ed-9949-a30143d02142-kube-api-access-tdbm6\") pod \"premium-w0-b0573c-kserve-55c76788d4-89gmm\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") " pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.637656 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.637632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:55:54.909531 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:54.909509    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"]
Sep 12 02:55:54.911564 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:55:54.911533    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7a71477_fd5b_47ed_9949_a30143d02142.slice/crio-93640c53817c970e1c233f020db33388e794728fad01fcf8e65831736f0f14dc WatchSource:0}: Error finding container 93640c53817c970e1c233f020db33388e794728fad01fcf8e65831736f0f14dc: Status 404 returned error can't find the container with id 93640c53817c970e1c233f020db33388e794728fad01fcf8e65831736f0f14dc
Sep 12 02:55:55.656783 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:55.656750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm" event={"ID":"b7a71477-fd5b-47ed-9949-a30143d02142","Type":"ContainerStarted","Data":"aca946bbd79525356307e625adf408addaec93801c5145bc0ed532b0588ac25e"}
Sep 12 02:55:55.656783 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:55.656790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm" event={"ID":"b7a71477-fd5b-47ed-9949-a30143d02142","Type":"ContainerStarted","Data":"93640c53817c970e1c233f020db33388e794728fad01fcf8e65831736f0f14dc"}
Sep 12 02:55:55.677434 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:55:55.677373    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm" podStartSLOduration=1.677356336 podStartE2EDuration="1.677356336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:55:55.675226007 +0000 UTC m=+1194.943259655" watchObservedRunningTime="2026-09-12 02:55:55.677356336 +0000 UTC m=+1194.945389984"
Sep 12 02:56:04.638652 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:04.638615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:56:04.639164 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:04.638668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:56:04.651271 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:04.651249    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:56:04.712998 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:04.712972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 02:56:10.252746 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.252710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:56:10.416183 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.416149    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"]
Sep 12 02:56:10.420935 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.420915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.423680 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.423663    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"premium-w4-c6190b-kserve-self-signed-certs\""
Sep 12 02:56:10.429525 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.429502    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"]
Sep 12 02:56:10.497024 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.496982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-model-cache\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.497024 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.497028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-dshm\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.497265 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.497087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-home\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.497306 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.497277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m78r\" (UniqueName: \"kubernetes.io/projected/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-kube-api-access-6m78r\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.497351 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.497314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tmp-dir\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.497351 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.497344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tls-certs\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-model-cache\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598343 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-dshm\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598580 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-home\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598580 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m78r\" (UniqueName: \"kubernetes.io/projected/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-kube-api-access-6m78r\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598580 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tmp-dir\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598580 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tls-certs\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598805 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-model-cache\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.598805 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-home\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.599002 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.598970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tmp-dir\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.600761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.600736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-dshm\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.601532 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.601507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tls-certs\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.606645 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.606625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m78r\" (UniqueName: \"kubernetes.io/projected/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-kube-api-access-6m78r\") pod \"premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") " pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:10.730802 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:10.730769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:11.005001 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:11.004963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"]
Sep 12 02:56:11.007001 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:56:11.006972    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c5751d3_515c_428b_9c76_cbdc7c7e4a12.slice/crio-4432f72e124a0c493e8e5e2dcfecba69d990b37bd4249c51797055a4b51cf560 WatchSource:0}: Error finding container 4432f72e124a0c493e8e5e2dcfecba69d990b37bd4249c51797055a4b51cf560: Status 404 returned error can't find the container with id 4432f72e124a0c493e8e5e2dcfecba69d990b37bd4249c51797055a4b51cf560
Sep 12 02:56:11.740756 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:11.740720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2" event={"ID":"6c5751d3-515c-428b-9c76-cbdc7c7e4a12","Type":"ContainerStarted","Data":"b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc"}
Sep 12 02:56:11.741166 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:11.740764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2" event={"ID":"6c5751d3-515c-428b-9c76-cbdc7c7e4a12","Type":"ContainerStarted","Data":"4432f72e124a0c493e8e5e2dcfecba69d990b37bd4249c51797055a4b51cf560"}
Sep 12 02:56:11.758238 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:11.758167    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2" podStartSLOduration=1.758151485 podStartE2EDuration="1.758151485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:56:11.756644368 +0000 UTC m=+1211.024678017" watchObservedRunningTime="2026-09-12 02:56:11.758151485 +0000 UTC m=+1211.026185132"
Sep 12 02:56:18.517610 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.517576    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"]
Sep 12 02:56:18.521790 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.521769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.524392 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.524369    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f93121\"/\"openshift-service-ca.crt\""
Sep 12 02:56:18.525368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.525346    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f93121\"/\"default-dockercfg-8hnmr\""
Sep 12 02:56:18.525368 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.525357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f93121\"/\"test-model-f93121-kserve-self-signed-certs\""
Sep 12 02:56:18.525552 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.525346    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f93121\"/\"kube-root-ca.crt\""
Sep 12 02:56:18.531150 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.531130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"]
Sep 12 02:56:18.673435 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.673404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-dshm\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.673585 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.673440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdv66\" (UniqueName: \"kubernetes.io/projected/9473f2db-432a-445c-a46a-d6bd5680205b-kube-api-access-hdv66\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.673585 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.673478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-home\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.673585 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.673525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-tmp-dir\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.673585 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.673571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-model-cache\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.673787 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.673602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9473f2db-432a-445c-a46a-d6bd5680205b-tls-certs\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774145 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-tmp-dir\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774282 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-model-cache\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774282 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9473f2db-432a-445c-a46a-d6bd5680205b-tls-certs\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774282 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-dshm\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774282 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdv66\" (UniqueName: \"kubernetes.io/projected/9473f2db-432a-445c-a46a-d6bd5680205b-kube-api-access-hdv66\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-home\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-tmp-dir\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774644 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-model-cache\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.774727 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.774709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-home\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.776727 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.776698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-dshm\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.776878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.776860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9473f2db-432a-445c-a46a-d6bd5680205b-tls-certs\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.782010 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.781987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdv66\" (UniqueName: \"kubernetes.io/projected/9473f2db-432a-445c-a46a-d6bd5680205b-kube-api-access-hdv66\") pod \"test-model-f93121-kserve-f58f4cccc-fgczq\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") " pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:18.832861 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:18.832835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:19.105823 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:19.105748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"]
Sep 12 02:56:19.109764 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:56:19.109735    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9473f2db_432a_445c_a46a_d6bd5680205b.slice/crio-2c2650bfad12b8b7e6e3b3afd650510b7c01e79499d65bc6c20aff8cd1d258ad WatchSource:0}: Error finding container 2c2650bfad12b8b7e6e3b3afd650510b7c01e79499d65bc6c20aff8cd1d258ad: Status 404 returned error can't find the container with id 2c2650bfad12b8b7e6e3b3afd650510b7c01e79499d65bc6c20aff8cd1d258ad
Sep 12 02:56:19.783287 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:19.783254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq" event={"ID":"9473f2db-432a-445c-a46a-d6bd5680205b","Type":"ContainerStarted","Data":"c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716"}
Sep 12 02:56:19.783287 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:19.783294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq" event={"ID":"9473f2db-432a-445c-a46a-d6bd5680205b","Type":"ContainerStarted","Data":"2c2650bfad12b8b7e6e3b3afd650510b7c01e79499d65bc6c20aff8cd1d258ad"}
Sep 12 02:56:19.801429 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:19.801381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq" podStartSLOduration=1.8013648039999999 podStartE2EDuration="1.801364804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:56:19.799807884 +0000 UTC m=+1219.067841535" watchObservedRunningTime="2026-09-12 02:56:19.801364804 +0000 UTC m=+1219.069398451"
Sep 12 02:56:20.731221 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:20.731183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:20.731420 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:20.731240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:20.744849 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:20.744821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:20.799839 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:20.799809    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 02:56:28.833417 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:28.833381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:28.833993 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:28.833429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:28.846373 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:28.846350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:28.857427 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:28.857406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:56:29.553674 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:29.553639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:56:51.540794 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.540756    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"]
Sep 12 02:56:51.544332 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.544303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.555751 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.555732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-662d15cb-serving-cert\""
Sep 12 02:56:51.649151 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.649120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"]
Sep 12 02:56:51.676127 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.676084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz52x\" (UniqueName: \"kubernetes.io/projected/b6d8144a-d965-4e2e-afd3-48a6742b490b-kube-api-access-lz52x\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.676278 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.676148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-metrics-certs\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.676278 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.676175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d8144a-d965-4e2e-afd3-48a6742b490b-openshift-service-ca\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.676396 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.676276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-maas-api-tls\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.777744 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.777714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-maas-api-tls\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.777885 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.777769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz52x\" (UniqueName: \"kubernetes.io/projected/b6d8144a-d965-4e2e-afd3-48a6742b490b-kube-api-access-lz52x\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.777885 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.777793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-metrics-certs\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.777885 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.777811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d8144a-d965-4e2e-afd3-48a6742b490b-openshift-service-ca\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.778540 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.778512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d8144a-d965-4e2e-afd3-48a6742b490b-openshift-service-ca\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.780366 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.780343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-maas-api-tls\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.780456 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.780401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-metrics-certs\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.789130 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.789076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz52x\" (UniqueName: \"kubernetes.io/projected/b6d8144a-d965-4e2e-afd3-48a6742b490b-kube-api-access-lz52x\") pod \"maas-api-e2e-mt-662d15cb-7676746654-qt9hv\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:51.856178 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:51.856086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:52.130902 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:52.130857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"]
Sep 12 02:56:52.133377 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:56:52.133349    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d8144a_d965_4e2e_afd3_48a6742b490b.slice/crio-53c985cbec4e5fb99858d74ed35fe4d3019f4a5e7efbae5b85b7afe76260ed39 WatchSource:0}: Error finding container 53c985cbec4e5fb99858d74ed35fe4d3019f4a5e7efbae5b85b7afe76260ed39: Status 404 returned error can't find the container with id 53c985cbec4e5fb99858d74ed35fe4d3019f4a5e7efbae5b85b7afe76260ed39
Sep 12 02:56:52.971024 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:52.970995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv" event={"ID":"b6d8144a-d965-4e2e-afd3-48a6742b490b","Type":"ContainerStarted","Data":"9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e"}
Sep 12 02:56:52.971416 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:52.971045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv" event={"ID":"b6d8144a-d965-4e2e-afd3-48a6742b490b","Type":"ContainerStarted","Data":"53c985cbec4e5fb99858d74ed35fe4d3019f4a5e7efbae5b85b7afe76260ed39"}
Sep 12 02:56:52.971416 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:52.971155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:56:52.989262 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:52.989219    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv" podStartSLOduration=1.607317572 podStartE2EDuration="1.989204939s" totalImagesPullingTime="381.887367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:56:52.987716585 +0000 UTC m=+1252.255750234" watchObservedRunningTime="2026-09-12 02:56:52.989204939 +0000 UTC m=+1252.257238585"
Sep 12 02:56:58.982046 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:56:58.982017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:57:01.450753 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:01.450721    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:57:32.355313 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:32.355278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:57:36.480245 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.480216    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"]
Sep 12 02:57:36.480746 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.480413    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv" podUID="b6d8144a-d965-4e2e-afd3-48a6742b490b" containerName="maas-api" containerID="cri-o://9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e" gracePeriod=30
Sep 12 02:57:36.731567 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.731511    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:57:36.800252 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.800222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz52x\" (UniqueName: \"kubernetes.io/projected/b6d8144a-d965-4e2e-afd3-48a6742b490b-kube-api-access-lz52x\") pod \"b6d8144a-d965-4e2e-afd3-48a6742b490b\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") "
Sep 12 02:57:36.800418 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.800321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d8144a-d965-4e2e-afd3-48a6742b490b-openshift-service-ca\") pod \"b6d8144a-d965-4e2e-afd3-48a6742b490b\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") "
Sep 12 02:57:36.800418 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.800354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-maas-api-tls\") pod \"b6d8144a-d965-4e2e-afd3-48a6742b490b\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") "
Sep 12 02:57:36.800418 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.800387    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-metrics-certs\") pod \"b6d8144a-d965-4e2e-afd3-48a6742b490b\" (UID: \"b6d8144a-d965-4e2e-afd3-48a6742b490b\") "
Sep 12 02:57:36.800780 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.800757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d8144a-d965-4e2e-afd3-48a6742b490b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6d8144a-d965-4e2e-afd3-48a6742b490b" (UID: "b6d8144a-d965-4e2e-afd3-48a6742b490b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:57:36.802522 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.802497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d8144a-d965-4e2e-afd3-48a6742b490b-kube-api-access-lz52x" (OuterVolumeSpecName: "kube-api-access-lz52x") pod "b6d8144a-d965-4e2e-afd3-48a6742b490b" (UID: "b6d8144a-d965-4e2e-afd3-48a6742b490b"). InnerVolumeSpecName "kube-api-access-lz52x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:57:36.802676 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.802660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6d8144a-d965-4e2e-afd3-48a6742b490b" (UID: "b6d8144a-d965-4e2e-afd3-48a6742b490b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:57:36.802761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.802747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6d8144a-d965-4e2e-afd3-48a6742b490b" (UID: "b6d8144a-d965-4e2e-afd3-48a6742b490b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:57:36.901851 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.901817    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d8144a-d965-4e2e-afd3-48a6742b490b-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:36.901851 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.901844    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:36.901851 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.901853    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d8144a-d965-4e2e-afd3-48a6742b490b-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:36.902067 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:36.901861    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz52x\" (UniqueName: \"kubernetes.io/projected/b6d8144a-d965-4e2e-afd3-48a6742b490b-kube-api-access-lz52x\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:37.221456 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.221418    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6d8144a-d965-4e2e-afd3-48a6742b490b" containerID="9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e" exitCode=0
Sep 12 02:57:37.221668 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.221465    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"
Sep 12 02:57:37.221668 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.221512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv" event={"ID":"b6d8144a-d965-4e2e-afd3-48a6742b490b","Type":"ContainerDied","Data":"9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e"}
Sep 12 02:57:37.221668 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.221571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv" event={"ID":"b6d8144a-d965-4e2e-afd3-48a6742b490b","Type":"ContainerDied","Data":"53c985cbec4e5fb99858d74ed35fe4d3019f4a5e7efbae5b85b7afe76260ed39"}
Sep 12 02:57:37.221668 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.221602    2579 scope.go:117] "RemoveContainer" containerID="9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e"
Sep 12 02:57:37.237587 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.237563    2579 scope.go:117] "RemoveContainer" containerID="9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e"
Sep 12 02:57:37.237899 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:57:37.237879    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e\": container with ID starting with 9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e not found: ID does not exist" containerID="9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e"
Sep 12 02:57:37.237951 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.237908    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e"} err="failed to get container status \"9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e\": rpc error: code = NotFound desc = could not find container \"9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e\": container with ID starting with 9f5faae8e1d99bd9e8872898c424e7904d42988317438295cc3dbe3fe6f27b6e not found: ID does not exist"
Sep 12 02:57:37.253145 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.253114    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"]
Sep 12 02:57:37.255658 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:37.255638    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-662d15cb-7676746654-qt9hv"]
Sep 12 02:57:39.249565 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:39.249534    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d8144a-d965-4e2e-afd3-48a6742b490b" path="/var/lib/kubelet/pods/b6d8144a-d965-4e2e-afd3-48a6742b490b/volumes"
Sep 12 02:57:41.566581 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.566551    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"]
Sep 12 02:57:41.567017 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.566749    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl" podUID="55c25822-2f10-4212-b2c5-e6b630607e6d" containerName="maas-api" containerID="cri-o://b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4" gracePeriod=30
Sep 12 02:57:41.612686 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.612645    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl" podUID="55c25822-2f10-4212-b2c5-e6b630607e6d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.49:8443/health\": dial tcp 10.132.0.49:8443: connect: connection refused"
Sep 12 02:57:41.826470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.826405    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:57:41.951706 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.951655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9w9f\" (UniqueName: \"kubernetes.io/projected/55c25822-2f10-4212-b2c5-e6b630607e6d-kube-api-access-f9w9f\") pod \"55c25822-2f10-4212-b2c5-e6b630607e6d\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") "
Sep 12 02:57:41.951886 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.951713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-maas-api-tls\") pod \"55c25822-2f10-4212-b2c5-e6b630607e6d\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") "
Sep 12 02:57:41.951886 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.951746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c25822-2f10-4212-b2c5-e6b630607e6d-openshift-service-ca\") pod \"55c25822-2f10-4212-b2c5-e6b630607e6d\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") "
Sep 12 02:57:41.952009 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.951865    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-metrics-certs\") pod \"55c25822-2f10-4212-b2c5-e6b630607e6d\" (UID: \"55c25822-2f10-4212-b2c5-e6b630607e6d\") "
Sep 12 02:57:41.952675 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.952076    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c25822-2f10-4212-b2c5-e6b630607e6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55c25822-2f10-4212-b2c5-e6b630607e6d" (UID: "55c25822-2f10-4212-b2c5-e6b630607e6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:57:41.952675 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.952372    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c25822-2f10-4212-b2c5-e6b630607e6d-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:41.954447 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.954425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55c25822-2f10-4212-b2c5-e6b630607e6d" (UID: "55c25822-2f10-4212-b2c5-e6b630607e6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:57:41.954617 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.954594    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55c25822-2f10-4212-b2c5-e6b630607e6d" (UID: "55c25822-2f10-4212-b2c5-e6b630607e6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:57:41.954888 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:41.954867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c25822-2f10-4212-b2c5-e6b630607e6d-kube-api-access-f9w9f" (OuterVolumeSpecName: "kube-api-access-f9w9f") pod "55c25822-2f10-4212-b2c5-e6b630607e6d" (UID: "55c25822-2f10-4212-b2c5-e6b630607e6d"). InnerVolumeSpecName "kube-api-access-f9w9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:57:42.053734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.053710    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9w9f\" (UniqueName: \"kubernetes.io/projected/55c25822-2f10-4212-b2c5-e6b630607e6d-kube-api-access-f9w9f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:42.053734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.053732    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:42.053734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.053741    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c25822-2f10-4212-b2c5-e6b630607e6d-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:57:42.249313 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.249277    2579 generic.go:356] "Generic (PLEG): container finished" podID="55c25822-2f10-4212-b2c5-e6b630607e6d" containerID="b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4" exitCode=0
Sep 12 02:57:42.249472 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.249317    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"
Sep 12 02:57:42.249472 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.249357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl" event={"ID":"55c25822-2f10-4212-b2c5-e6b630607e6d","Type":"ContainerDied","Data":"b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4"}
Sep 12 02:57:42.249472 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.249406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl" event={"ID":"55c25822-2f10-4212-b2c5-e6b630607e6d","Type":"ContainerDied","Data":"6fd93177ae6616317b80f3b98bc7e1bff1339770dc5dceb3be53a14de83d810f"}
Sep 12 02:57:42.249472 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.249438    2579 scope.go:117] "RemoveContainer" containerID="b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4"
Sep 12 02:57:42.259039 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.259022    2579 scope.go:117] "RemoveContainer" containerID="b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4"
Sep 12 02:57:42.259338 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:57:42.259318    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4\": container with ID starting with b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4 not found: ID does not exist" containerID="b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4"
Sep 12 02:57:42.259410 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.259346    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4"} err="failed to get container status \"b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4\": rpc error: code = NotFound desc = could not find container \"b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4\": container with ID starting with b60ed8255af4f1ca75140d985cf53383df6f1bf37db81b4d6565ce78835746b4 not found: ID does not exist"
Sep 12 02:57:42.270910 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.270892    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"]
Sep 12 02:57:42.274884 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:42.274865    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-450f2a-7875b96dbd-zdpwl"]
Sep 12 02:57:43.249434 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:57:43.249400    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c25822-2f10-4212-b2c5-e6b630607e6d" path="/var/lib/kubelet/pods/55c25822-2f10-4212-b2c5-e6b630607e6d/volumes"
Sep 12 02:58:03.054921 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.054888    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"]
Sep 12 02:58:03.055349 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.055333    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d8144a-d965-4e2e-afd3-48a6742b490b" containerName="maas-api"
Sep 12 02:58:03.055399 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.055351    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d8144a-d965-4e2e-afd3-48a6742b490b" containerName="maas-api"
Sep 12 02:58:03.055399 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.055368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c25822-2f10-4212-b2c5-e6b630607e6d" containerName="maas-api"
Sep 12 02:58:03.055399 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.055373    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c25822-2f10-4212-b2c5-e6b630607e6d" containerName="maas-api"
Sep 12 02:58:03.055491 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.055438    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c25822-2f10-4212-b2c5-e6b630607e6d" containerName="maas-api"
Sep 12 02:58:03.055491 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.055447    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d8144a-d965-4e2e-afd3-48a6742b490b" containerName="maas-api"
Sep 12 02:58:03.059792 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.059775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.062160 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.062137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-933bba-serving-cert\""
Sep 12 02:58:03.068931 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.068908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"]
Sep 12 02:58:03.155045 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.155014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7tm\" (UniqueName: \"kubernetes.io/projected/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-kube-api-access-gb7tm\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.155213 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.155067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.155213 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.155197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.155294 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.155235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.256446 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.256415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7tm\" (UniqueName: \"kubernetes.io/projected/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-kube-api-access-gb7tm\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.256623 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.256465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.256623 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.256508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.256623 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.256524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.257195 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.257168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.259223 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.259202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.259333 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.259312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.264922 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.264905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7tm\" (UniqueName: \"kubernetes.io/projected/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-kube-api-access-gb7tm\") pod \"maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.371212 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.371147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:03.642225 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.642202    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"]
Sep 12 02:58:03.644430 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:58:03.644399    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bac08d_5a46_48c4_b1e0_3b433ebf27ed.slice/crio-5505cae9624e47e5953d4f4c14208881fec4d53697fcc7a87c7f954228b44242 WatchSource:0}: Error finding container 5505cae9624e47e5953d4f4c14208881fec4d53697fcc7a87c7f954228b44242: Status 404 returned error can't find the container with id 5505cae9624e47e5953d4f4c14208881fec4d53697fcc7a87c7f954228b44242
Sep 12 02:58:03.645941 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:03.645924    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:58:04.371047 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:04.370969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6" event={"ID":"58bac08d-5a46-48c4-b1e0-3b433ebf27ed","Type":"ContainerStarted","Data":"80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28"}
Sep 12 02:58:04.371047 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:04.371020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6" event={"ID":"58bac08d-5a46-48c4-b1e0-3b433ebf27ed","Type":"ContainerStarted","Data":"5505cae9624e47e5953d4f4c14208881fec4d53697fcc7a87c7f954228b44242"}
Sep 12 02:58:04.371452 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:04.371126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:04.390320 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:04.390275    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6" podStartSLOduration=0.942147283 podStartE2EDuration="1.390259869s" totalImagesPullingTime="448.112586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:58:04.388433182 +0000 UTC m=+1323.656466829" watchObservedRunningTime="2026-09-12 02:58:04.390259869 +0000 UTC m=+1323.658293516"
Sep 12 02:58:04.753491 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:04.753462    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:58:10.381480 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:10.381450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 02:58:14.403206 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.403171    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"]
Sep 12 02:58:14.407007 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.406987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.409160 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.409138    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-697fd4\"/\"default-dockercfg-9wd9x\""
Sep 12 02:58:14.410115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.410080    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-697fd4\"/\"openshift-service-ca.crt\""
Sep 12 02:58:14.410215 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.410085    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-697fd4\"/\"kube-root-ca.crt\""
Sep 12 02:58:14.410215 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.410192    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-697fd4\"/\"noten-model-697fd4-kserve-self-signed-certs\""
Sep 12 02:58:14.416070 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.416049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"]
Sep 12 02:58:14.570927 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.570894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcv8w\" (UniqueName: \"kubernetes.io/projected/f21a701c-400d-4608-aaf6-e5581f1b059f-kube-api-access-lcv8w\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.571115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.570939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-home\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.571115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.570992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21a701c-400d-4608-aaf6-e5581f1b059f-tls-certs\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.571115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.571030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-model-cache\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.571115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.571071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-dshm\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.571282 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.571152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-tmp-dir\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.672550 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-home\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.672550 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21a701c-400d-4608-aaf6-e5581f1b059f-tls-certs\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.672550 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-model-cache\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.672785 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-dshm\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.672785 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-tmp-dir\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.672785 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcv8w\" (UniqueName: \"kubernetes.io/projected/f21a701c-400d-4608-aaf6-e5581f1b059f-kube-api-access-lcv8w\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.672947 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-home\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.673115 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.672952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-model-cache\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.673181 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.673154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-tmp-dir\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.675498 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.675473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-dshm\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.675744 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.675726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21a701c-400d-4608-aaf6-e5581f1b059f-tls-certs\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.679829 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.679806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcv8w\" (UniqueName: \"kubernetes.io/projected/f21a701c-400d-4608-aaf6-e5581f1b059f-kube-api-access-lcv8w\") pod \"noten-model-697fd4-kserve-67b8d49555-hbs79\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") " pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.717792 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.717773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:14.991510 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:14.991414    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"]
Sep 12 02:58:14.994533 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:58:14.994495    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf21a701c_400d_4608_aaf6_e5581f1b059f.slice/crio-9758e623f85e660a4537fa7a0aba2c76eafe1f0bc0aa27e1a33fc9da289c51f7 WatchSource:0}: Error finding container 9758e623f85e660a4537fa7a0aba2c76eafe1f0bc0aa27e1a33fc9da289c51f7: Status 404 returned error can't find the container with id 9758e623f85e660a4537fa7a0aba2c76eafe1f0bc0aa27e1a33fc9da289c51f7
Sep 12 02:58:15.438032 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:15.437999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79" event={"ID":"f21a701c-400d-4608-aaf6-e5581f1b059f","Type":"ContainerStarted","Data":"9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483"}
Sep 12 02:58:15.438444 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:15.438047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79" event={"ID":"f21a701c-400d-4608-aaf6-e5581f1b059f","Type":"ContainerStarted","Data":"9758e623f85e660a4537fa7a0aba2c76eafe1f0bc0aa27e1a33fc9da289c51f7"}
Sep 12 02:58:15.459634 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:15.459588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79" podStartSLOduration=1.459572216 podStartE2EDuration="1.459572216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:58:15.458265085 +0000 UTC m=+1334.726298733" watchObservedRunningTime="2026-09-12 02:58:15.459572216 +0000 UTC m=+1334.727605862"
Sep 12 02:58:24.718878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:24.718801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:24.718878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:24.718849    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:24.731119 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:24.731074    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:25.504025 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:25.503992    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:58:27.841657 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:27.841624    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:58:53.406913 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:58:53.406878    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"]
Sep 12 02:59:04.248470 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:04.248435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:59:04.959445 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:04.959414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-a21fe5"]
Sep 12 02:59:04.962991 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:04.962973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:04.965230 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:04.965207    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sww9\""
Sep 12 02:59:04.968807 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:04.968781    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-a21fe5"]
Sep 12 02:59:04.974040 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:04.974019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgpn\" (UniqueName: \"kubernetes.io/projected/78b06226-d723-4939-b6ac-6b913812878c-kube-api-access-kfgpn\") pod \"test-curl-7879-a21fe5\" (UID: \"78b06226-d723-4939-b6ac-6b913812878c\") " pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:05.074578 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:05.074548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgpn\" (UniqueName: \"kubernetes.io/projected/78b06226-d723-4939-b6ac-6b913812878c-kube-api-access-kfgpn\") pod \"test-curl-7879-a21fe5\" (UID: \"78b06226-d723-4939-b6ac-6b913812878c\") " pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:05.082208 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:05.082184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgpn\" (UniqueName: \"kubernetes.io/projected/78b06226-d723-4939-b6ac-6b913812878c-kube-api-access-kfgpn\") pod \"test-curl-7879-a21fe5\" (UID: \"78b06226-d723-4939-b6ac-6b913812878c\") " pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:05.273831 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:05.273758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:05.545366 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:05.545341    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-a21fe5"]
Sep 12 02:59:05.547778 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:59:05.547755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78b06226_d723_4939_b6ac_6b913812878c.slice/crio-175db43b5ca1d8a85faa02143983b7f9fd3e09457a9c9f9396b4c7d4e765169e WatchSource:0}: Error finding container 175db43b5ca1d8a85faa02143983b7f9fd3e09457a9c9f9396b4c7d4e765169e: Status 404 returned error can't find the container with id 175db43b5ca1d8a85faa02143983b7f9fd3e09457a9c9f9396b4c7d4e765169e
Sep 12 02:59:05.734142 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:05.734111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-a21fe5" event={"ID":"78b06226-d723-4939-b6ac-6b913812878c","Type":"ContainerStarted","Data":"2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc"}
Sep 12 02:59:05.734142 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:05.734154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-a21fe5" event={"ID":"78b06226-d723-4939-b6ac-6b913812878c","Type":"ContainerStarted","Data":"175db43b5ca1d8a85faa02143983b7f9fd3e09457a9c9f9396b4c7d4e765169e"}
Sep 12 02:59:05.748620 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:05.748576    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7879-a21fe5" podStartSLOduration=1.748561364 podStartE2EDuration="1.748561364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:05.746536068 +0000 UTC m=+1385.014569715" watchObservedRunningTime="2026-09-12 02:59:05.748561364 +0000 UTC m=+1385.016595011"
Sep 12 02:59:06.343524 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.343492    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-a21fe5"]
Sep 12 02:59:06.347427 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.347399    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-a21fe5"]
Sep 12 02:59:06.471236 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.471208    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-e22f5a"]
Sep 12 02:59:06.471595 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.471582    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b06226-d723-4939-b6ac-6b913812878c" containerName="test-curl-7879-a21fe5"
Sep 12 02:59:06.471639 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.471597    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b06226-d723-4939-b6ac-6b913812878c" containerName="test-curl-7879-a21fe5"
Sep 12 02:59:06.471694 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.471686    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b06226-d723-4939-b6ac-6b913812878c" containerName="test-curl-7879-a21fe5"
Sep 12 02:59:06.474876 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.474859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:06.480261 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.480234    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-e22f5a"]
Sep 12 02:59:06.489347 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.489323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6prd\" (UniqueName: \"kubernetes.io/projected/93c2bfa0-c5c3-4af8-947e-7c2647039ccb-kube-api-access-w6prd\") pod \"test-curl-7879-e22f5a\" (UID: \"93c2bfa0-c5c3-4af8-947e-7c2647039ccb\") " pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:06.590559 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.590529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6prd\" (UniqueName: \"kubernetes.io/projected/93c2bfa0-c5c3-4af8-947e-7c2647039ccb-kube-api-access-w6prd\") pod \"test-curl-7879-e22f5a\" (UID: \"93c2bfa0-c5c3-4af8-947e-7c2647039ccb\") " pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:06.597750 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.597692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6prd\" (UniqueName: \"kubernetes.io/projected/93c2bfa0-c5c3-4af8-947e-7c2647039ccb-kube-api-access-w6prd\") pod \"test-curl-7879-e22f5a\" (UID: \"93c2bfa0-c5c3-4af8-947e-7c2647039ccb\") " pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:06.739400 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:06.739379    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7879-a21fe5" container="test-curl-7879-a21fe5"
Sep 12 02:59:06.785862 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:06.785839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:07.053784 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:07.053763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-e22f5a"]
Sep 12 02:59:07.055835 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:59:07.055804    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93c2bfa0_c5c3_4af8_947e_7c2647039ccb.slice/crio-284bf357b05ae5960eb40b97ac876ed2e9163bf975ec8faaafdaf72015a36e21 WatchSource:0}: Error finding container 284bf357b05ae5960eb40b97ac876ed2e9163bf975ec8faaafdaf72015a36e21: Status 404 returned error can't find the container with id 284bf357b05ae5960eb40b97ac876ed2e9163bf975ec8faaafdaf72015a36e21
Sep 12 02:59:07.746324 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:07.746288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-e22f5a" event={"ID":"93c2bfa0-c5c3-4af8-947e-7c2647039ccb","Type":"ContainerStarted","Data":"e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873"}
Sep 12 02:59:07.746789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:07.746336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-e22f5a" event={"ID":"93c2bfa0-c5c3-4af8-947e-7c2647039ccb","Type":"ContainerStarted","Data":"284bf357b05ae5960eb40b97ac876ed2e9163bf975ec8faaafdaf72015a36e21"}
Sep 12 02:59:07.746789 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:07.746414    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7879-a21fe5" podUID="78b06226-d723-4939-b6ac-6b913812878c" containerName="test-curl-7879-a21fe5" containerID="cri-o://2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc" gracePeriod=2
Sep 12 02:59:07.758963 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:07.758919    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7879-e22f5a" podStartSLOduration=1.758905377 podStartE2EDuration="1.758905377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:07.75808143 +0000 UTC m=+1387.026115090" watchObservedRunningTime="2026-09-12 02:59:07.758905377 +0000 UTC m=+1387.026939024"
Sep 12 02:59:08.359001 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.358972    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-e22f5a"]
Sep 12 02:59:08.364753 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.364723    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-e22f5a"]
Sep 12 02:59:08.428087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.428007    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79" podUID="f21a701c-400d-4608-aaf6-e5581f1b059f" containerName="main" containerID="cri-o://9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483" gracePeriod=45
Sep 12 02:59:08.482081 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.482055    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-4e2bbf"]
Sep 12 02:59:08.482669 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.482654    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c2bfa0-c5c3-4af8-947e-7c2647039ccb" containerName="test-curl-7879-e22f5a"
Sep 12 02:59:08.482719 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.482671    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c2bfa0-c5c3-4af8-947e-7c2647039ccb" containerName="test-curl-7879-e22f5a"
Sep 12 02:59:08.482755 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.482749    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c2bfa0-c5c3-4af8-947e-7c2647039ccb" containerName="test-curl-7879-e22f5a"
Sep 12 02:59:08.485761 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.485746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:08.488979 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.488957    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-4e2bbf"]
Sep 12 02:59:08.512050 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.512021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv6vw\" (UniqueName: \"kubernetes.io/projected/09d68593-e5a4-45b1-a442-81f56628e25a-kube-api-access-hv6vw\") pod \"test-curl-7879-4e2bbf\" (UID: \"09d68593-e5a4-45b1-a442-81f56628e25a\") " pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:08.612843 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.612817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv6vw\" (UniqueName: \"kubernetes.io/projected/09d68593-e5a4-45b1-a442-81f56628e25a-kube-api-access-hv6vw\") pod \"test-curl-7879-4e2bbf\" (UID: \"09d68593-e5a4-45b1-a442-81f56628e25a\") " pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:08.621015 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.620990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv6vw\" (UniqueName: \"kubernetes.io/projected/09d68593-e5a4-45b1-a442-81f56628e25a-kube-api-access-hv6vw\") pod \"test-curl-7879-4e2bbf\" (UID: \"09d68593-e5a4-45b1-a442-81f56628e25a\") " pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:08.671856 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.671836    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:59:08.713839 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.713802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-dshm\") pod \"f21a701c-400d-4608-aaf6-e5581f1b059f\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") "
Sep 12 02:59:08.714011 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.713843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcv8w\" (UniqueName: \"kubernetes.io/projected/f21a701c-400d-4608-aaf6-e5581f1b059f-kube-api-access-lcv8w\") pod \"f21a701c-400d-4608-aaf6-e5581f1b059f\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") "
Sep 12 02:59:08.714011 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.713945    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-tmp-dir\") pod \"f21a701c-400d-4608-aaf6-e5581f1b059f\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") "
Sep 12 02:59:08.714172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-home\") pod \"f21a701c-400d-4608-aaf6-e5581f1b059f\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") "
Sep 12 02:59:08.714172 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21a701c-400d-4608-aaf6-e5581f1b059f-tls-certs\") pod \"f21a701c-400d-4608-aaf6-e5581f1b059f\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") "
Sep 12 02:59:08.714286 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714184    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-model-cache\") pod \"f21a701c-400d-4608-aaf6-e5581f1b059f\" (UID: \"f21a701c-400d-4608-aaf6-e5581f1b059f\") "
Sep 12 02:59:08.714286 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f21a701c-400d-4608-aaf6-e5581f1b059f" (UID: "f21a701c-400d-4608-aaf6-e5581f1b059f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:08.714388 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714325    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-home" (OuterVolumeSpecName: "home") pod "f21a701c-400d-4608-aaf6-e5581f1b059f" (UID: "f21a701c-400d-4608-aaf6-e5581f1b059f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:08.714527 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714505    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f21a701c-400d-4608-aaf6-e5581f1b059f" (UID: "f21a701c-400d-4608-aaf6-e5581f1b059f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:08.714647 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714631    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:08.714719 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714651    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:08.714719 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.714664    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:08.716660 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.716613    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-dshm" (OuterVolumeSpecName: "dshm") pod "f21a701c-400d-4608-aaf6-e5581f1b059f" (UID: "f21a701c-400d-4608-aaf6-e5581f1b059f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:08.717022 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.716958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21a701c-400d-4608-aaf6-e5581f1b059f-kube-api-access-lcv8w" (OuterVolumeSpecName: "kube-api-access-lcv8w") pod "f21a701c-400d-4608-aaf6-e5581f1b059f" (UID: "f21a701c-400d-4608-aaf6-e5581f1b059f"). InnerVolumeSpecName "kube-api-access-lcv8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:08.717220 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.717174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21a701c-400d-4608-aaf6-e5581f1b059f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f21a701c-400d-4608-aaf6-e5581f1b059f" (UID: "f21a701c-400d-4608-aaf6-e5581f1b059f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:59:08.753453 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.753429    2579 generic.go:356] "Generic (PLEG): container finished" podID="f21a701c-400d-4608-aaf6-e5581f1b059f" containerID="9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483" exitCode=0
Sep 12 02:59:08.753828 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.753474    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"
Sep 12 02:59:08.753828 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.753504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79" event={"ID":"f21a701c-400d-4608-aaf6-e5581f1b059f","Type":"ContainerDied","Data":"9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483"}
Sep 12 02:59:08.753828 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.753536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79" event={"ID":"f21a701c-400d-4608-aaf6-e5581f1b059f","Type":"ContainerDied","Data":"9758e623f85e660a4537fa7a0aba2c76eafe1f0bc0aa27e1a33fc9da289c51f7"}
Sep 12 02:59:08.753828 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.753552    2579 scope.go:117] "RemoveContainer" containerID="9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483"
Sep 12 02:59:08.754008 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:08.753884    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7879-e22f5a" container="test-curl-7879-e22f5a"
Sep 12 02:59:08.762770 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.762754    2579 scope.go:117] "RemoveContainer" containerID="9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483"
Sep 12 02:59:08.763037 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:08.763021    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483\": container with ID starting with 9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483 not found: ID does not exist" containerID="9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483"
Sep 12 02:59:08.763087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.763043    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483"} err="failed to get container status \"9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483\": rpc error: code = NotFound desc = could not find container \"9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483\": container with ID starting with 9dd9bfde1fa0174f05fd12e2f9c60262335a2e9f3f60ae7959add33167107483 not found: ID does not exist"
Sep 12 02:59:08.774058 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.774027    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"]
Sep 12 02:59:08.777153 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.777134    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-697fd4/noten-model-697fd4-kserve-67b8d49555-hbs79"]
Sep 12 02:59:08.798328 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.798307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:08.815775 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.815756    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21a701c-400d-4608-aaf6-e5581f1b059f-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:08.815775 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.815776    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21a701c-400d-4608-aaf6-e5581f1b059f-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:08.815898 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:08.815786    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcv8w\" (UniqueName: \"kubernetes.io/projected/f21a701c-400d-4608-aaf6-e5581f1b059f-kube-api-access-lcv8w\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:09.068300 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:09.068275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-4e2bbf"]
Sep 12 02:59:09.070203 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:59:09.070182    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d68593_e5a4_45b1_a442_81f56628e25a.slice/crio-6df7043ea47feeeede0b1bef8c048f8380e0df44ec146ec153f6c79d58fd1859 WatchSource:0}: Error finding container 6df7043ea47feeeede0b1bef8c048f8380e0df44ec146ec153f6c79d58fd1859: Status 404 returned error can't find the container with id 6df7043ea47feeeede0b1bef8c048f8380e0df44ec146ec153f6c79d58fd1859
Sep 12 02:59:09.249943 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:09.249909    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f21a701c-400d-4608-aaf6-e5581f1b059f" path="/var/lib/kubelet/pods/f21a701c-400d-4608-aaf6-e5581f1b059f/volumes"
Sep 12 02:59:09.763474 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:09.763439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-4e2bbf" event={"ID":"09d68593-e5a4-45b1-a442-81f56628e25a","Type":"ContainerStarted","Data":"f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350"}
Sep 12 02:59:09.763835 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:09.763492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-4e2bbf" event={"ID":"09d68593-e5a4-45b1-a442-81f56628e25a","Type":"ContainerStarted","Data":"6df7043ea47feeeede0b1bef8c048f8380e0df44ec146ec153f6c79d58fd1859"}
Sep 12 02:59:09.763835 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:09.763604    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7879-e22f5a" podUID="93c2bfa0-c5c3-4af8-947e-7c2647039ccb" containerName="test-curl-7879-e22f5a" containerID="cri-o://e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873" gracePeriod=2
Sep 12 02:59:09.932020 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:09.932000    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:10.026075 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.026016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfgpn\" (UniqueName: \"kubernetes.io/projected/78b06226-d723-4939-b6ac-6b913812878c-kube-api-access-kfgpn\") pod \"78b06226-d723-4939-b6ac-6b913812878c\" (UID: \"78b06226-d723-4939-b6ac-6b913812878c\") "
Sep 12 02:59:10.028030 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.028001    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b06226-d723-4939-b6ac-6b913812878c-kube-api-access-kfgpn" (OuterVolumeSpecName: "kube-api-access-kfgpn") pod "78b06226-d723-4939-b6ac-6b913812878c" (UID: "78b06226-d723-4939-b6ac-6b913812878c"). InnerVolumeSpecName "kube-api-access-kfgpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:10.127732 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.127705    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgpn\" (UniqueName: \"kubernetes.io/projected/78b06226-d723-4939-b6ac-6b913812878c-kube-api-access-kfgpn\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:10.348949 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.348844    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7879-4e2bbf" podStartSLOduration=2.348826512 podStartE2EDuration="2.348826512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:09.775693777 +0000 UTC m=+1389.043727425" watchObservedRunningTime="2026-09-12 02:59:10.348826512 +0000 UTC m=+1389.616860159"
Sep 12 02:59:10.349640 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.349623    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-4e2bbf"]
Sep 12 02:59:10.353265 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.353238    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-4e2bbf"]
Sep 12 02:59:10.477199 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.477166    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7879-6b7eb0"]
Sep 12 02:59:10.477578 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.477565    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d68593-e5a4-45b1-a442-81f56628e25a" containerName="test-curl-7879-4e2bbf"
Sep 12 02:59:10.477646 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.477580    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d68593-e5a4-45b1-a442-81f56628e25a" containerName="test-curl-7879-4e2bbf"
Sep 12 02:59:10.477646 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.477594    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21a701c-400d-4608-aaf6-e5581f1b059f" containerName="main"
Sep 12 02:59:10.477646 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.477599    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21a701c-400d-4608-aaf6-e5581f1b059f" containerName="main"
Sep 12 02:59:10.477752 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.477654    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d68593-e5a4-45b1-a442-81f56628e25a" containerName="test-curl-7879-4e2bbf"
Sep 12 02:59:10.477752 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.477663    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21a701c-400d-4608-aaf6-e5581f1b059f" containerName="main"
Sep 12 02:59:10.480688 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.480673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:10.484324 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.484296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-6b7eb0"]
Sep 12 02:59:10.532200 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.532172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqj4\" (UniqueName: \"kubernetes.io/projected/7ba8affc-3e8d-4bfa-9cd6-32ac0593a343-kube-api-access-sbqj4\") pod \"test-curl-7879-6b7eb0\" (UID: \"7ba8affc-3e8d-4bfa-9cd6-32ac0593a343\") " pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:10.633850 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.633762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqj4\" (UniqueName: \"kubernetes.io/projected/7ba8affc-3e8d-4bfa-9cd6-32ac0593a343-kube-api-access-sbqj4\") pod \"test-curl-7879-6b7eb0\" (UID: \"7ba8affc-3e8d-4bfa-9cd6-32ac0593a343\") " pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:10.641314 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.641283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqj4\" (UniqueName: \"kubernetes.io/projected/7ba8affc-3e8d-4bfa-9cd6-32ac0593a343-kube-api-access-sbqj4\") pod \"test-curl-7879-6b7eb0\" (UID: \"7ba8affc-3e8d-4bfa-9cd6-32ac0593a343\") " pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:10.770243 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.770206    2579 generic.go:356] "Generic (PLEG): container finished" podID="78b06226-d723-4939-b6ac-6b913812878c" containerID="2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc" exitCode=137
Sep 12 02:59:10.770243 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.770245    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:10.770721 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.770309    2579 scope.go:117] "RemoveContainer" containerID="2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc"
Sep 12 02:59:10.770721 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:10.770692    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7879-4e2bbf" container="test-curl-7879-4e2bbf"
Sep 12 02:59:10.782297 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.782273    2579 scope.go:117] "RemoveContainer" containerID="2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc"
Sep 12 02:59:10.782397 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:10.782257    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-a21fe5\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="78b06226-d723-4939-b6ac-6b913812878c" pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:10.782737 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:10.782715    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc\": container with ID starting with 2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc not found: ID does not exist" containerID="2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc"
Sep 12 02:59:10.782809 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.782752    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc"} err="failed to get container status \"2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc\": rpc error: code = NotFound desc = could not find container \"2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc\": container with ID starting with 2ad63f01900cec72dc11491d64c6af089715ce26186b325330f36e01db7349bc not found: ID does not exist"
Sep 12 02:59:10.793405 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:10.793387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:11.064422 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:11.064400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7879-6b7eb0"]
Sep 12 02:59:11.066950 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:59:11.066924    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ba8affc_3e8d_4bfa_9cd6_32ac0593a343.slice/crio-063f002a39da57db713b9903634e54f636a151fee8010d9321f0187fc6992d27 WatchSource:0}: Error finding container 063f002a39da57db713b9903634e54f636a151fee8010d9321f0187fc6992d27: Status 404 returned error can't find the container with id 063f002a39da57db713b9903634e54f636a151fee8010d9321f0187fc6992d27
Sep 12 02:59:11.249224 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:11.249191    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-a21fe5\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="78b06226-d723-4939-b6ac-6b913812878c" pod="opendatahub/test-curl-7879-a21fe5"
Sep 12 02:59:11.251595 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:11.251575    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b06226-d723-4939-b6ac-6b913812878c" path="/var/lib/kubelet/pods/78b06226-d723-4939-b6ac-6b913812878c/volumes"
Sep 12 02:59:11.776630 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:11.776601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-6b7eb0" event={"ID":"7ba8affc-3e8d-4bfa-9cd6-32ac0593a343","Type":"ContainerStarted","Data":"7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9"}
Sep 12 02:59:11.776976 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:11.776651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7879-6b7eb0" event={"ID":"7ba8affc-3e8d-4bfa-9cd6-32ac0593a343","Type":"ContainerStarted","Data":"063f002a39da57db713b9903634e54f636a151fee8010d9321f0187fc6992d27"}
Sep 12 02:59:11.777669 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:11.777622    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7879-4e2bbf" podUID="09d68593-e5a4-45b1-a442-81f56628e25a" containerName="test-curl-7879-4e2bbf" containerID="cri-o://f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350" gracePeriod=2
Sep 12 02:59:11.791076 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:11.791013    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7879-6b7eb0" podStartSLOduration=1.790994423 podStartE2EDuration="1.790994423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:11.7905387 +0000 UTC m=+1391.058572347" watchObservedRunningTime="2026-09-12 02:59:11.790994423 +0000 UTC m=+1391.059028071"
Sep 12 02:59:11.942418 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:11.942384    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:12.045964 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.045878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6prd\" (UniqueName: \"kubernetes.io/projected/93c2bfa0-c5c3-4af8-947e-7c2647039ccb-kube-api-access-w6prd\") pod \"93c2bfa0-c5c3-4af8-947e-7c2647039ccb\" (UID: \"93c2bfa0-c5c3-4af8-947e-7c2647039ccb\") "
Sep 12 02:59:12.047859 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.047832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c2bfa0-c5c3-4af8-947e-7c2647039ccb-kube-api-access-w6prd" (OuterVolumeSpecName: "kube-api-access-w6prd") pod "93c2bfa0-c5c3-4af8-947e-7c2647039ccb" (UID: "93c2bfa0-c5c3-4af8-947e-7c2647039ccb"). InnerVolumeSpecName "kube-api-access-w6prd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:12.146878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.146853    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6prd\" (UniqueName: \"kubernetes.io/projected/93c2bfa0-c5c3-4af8-947e-7c2647039ccb-kube-api-access-w6prd\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:12.372036 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.371953    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7879-6b7eb0"]
Sep 12 02:59:12.375577 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.375548    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7879-6b7eb0"]
Sep 12 02:59:12.784179 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.784136    2579 generic.go:356] "Generic (PLEG): container finished" podID="93c2bfa0-c5c3-4af8-947e-7c2647039ccb" containerID="e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873" exitCode=137
Sep 12 02:59:12.784179 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.784177    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:12.784647 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.784214    2579 scope.go:117] "RemoveContainer" containerID="e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873"
Sep 12 02:59:12.784647 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:12.784441    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7879-6b7eb0" container="test-curl-7879-6b7eb0"
Sep 12 02:59:12.794051 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:12.794029    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-e22f5a\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="93c2bfa0-c5c3-4af8-947e-7c2647039ccb" pod="opendatahub/test-curl-7879-e22f5a"
Sep 12 02:59:12.794996 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.794981    2579 scope.go:117] "RemoveContainer" containerID="e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873"
Sep 12 02:59:12.795301 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:12.795285    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873\": container with ID starting with e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873 not found: ID does not exist" containerID="e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873"
Sep 12 02:59:12.795373 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:12.795308    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873"} err="failed to get container status \"e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873\": rpc error: code = NotFound desc = could not find container \"e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873\": container with ID starting with e7476224bdc4ec584b6aaf9659904245043522ad71eaf0e5e0c6051c5bd3e873 not found: ID does not exist"
Sep 12 02:59:13.253514 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:13.253478    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c2bfa0-c5c3-4af8-947e-7c2647039ccb" path="/var/lib/kubelet/pods/93c2bfa0-c5c3-4af8-947e-7c2647039ccb/volumes"
Sep 12 02:59:13.790798 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:13.790767    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7879-6b7eb0" podUID="7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" containerName="test-curl-7879-6b7eb0" containerID="cri-o://7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9" gracePeriod=2
Sep 12 02:59:13.964037 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:13.964015    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:14.065500 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.065416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv6vw\" (UniqueName: \"kubernetes.io/projected/09d68593-e5a4-45b1-a442-81f56628e25a-kube-api-access-hv6vw\") pod \"09d68593-e5a4-45b1-a442-81f56628e25a\" (UID: \"09d68593-e5a4-45b1-a442-81f56628e25a\") "
Sep 12 02:59:14.067695 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.067674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d68593-e5a4-45b1-a442-81f56628e25a-kube-api-access-hv6vw" (OuterVolumeSpecName: "kube-api-access-hv6vw") pod "09d68593-e5a4-45b1-a442-81f56628e25a" (UID: "09d68593-e5a4-45b1-a442-81f56628e25a"). InnerVolumeSpecName "kube-api-access-hv6vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:14.166784 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.166749    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv6vw\" (UniqueName: \"kubernetes.io/projected/09d68593-e5a4-45b1-a442-81f56628e25a-kube-api-access-hv6vw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:14.798138 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.798084    2579 generic.go:356] "Generic (PLEG): container finished" podID="09d68593-e5a4-45b1-a442-81f56628e25a" containerID="f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350" exitCode=137
Sep 12 02:59:14.798138 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.798140    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:14.798636 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.798173    2579 scope.go:117] "RemoveContainer" containerID="f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350"
Sep 12 02:59:14.807704 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.807689    2579 scope.go:117] "RemoveContainer" containerID="f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350"
Sep 12 02:59:14.807963 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:14.807944    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350\": container with ID starting with f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350 not found: ID does not exist" containerID="f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350"
Sep 12 02:59:14.808008 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:14.807978    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350"} err="failed to get container status \"f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350\": rpc error: code = NotFound desc = could not find container \"f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350\": container with ID starting with f3c98b5c32de827cec795ba2afe231641e263064ebd4cbba44830e0c5cb84350 not found: ID does not exist"
Sep 12 02:59:14.808129 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:14.808111    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-4e2bbf\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="09d68593-e5a4-45b1-a442-81f56628e25a" pod="opendatahub/test-curl-7879-4e2bbf"
Sep 12 02:59:15.250380 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:15.250349    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d68593-e5a4-45b1-a442-81f56628e25a" path="/var/lib/kubelet/pods/09d68593-e5a4-45b1-a442-81f56628e25a/volumes"
Sep 12 02:59:15.962699 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:15.962672    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:16.082993 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.082924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbqj4\" (UniqueName: \"kubernetes.io/projected/7ba8affc-3e8d-4bfa-9cd6-32ac0593a343-kube-api-access-sbqj4\") pod \"7ba8affc-3e8d-4bfa-9cd6-32ac0593a343\" (UID: \"7ba8affc-3e8d-4bfa-9cd6-32ac0593a343\") "
Sep 12 02:59:16.084934 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.084905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba8affc-3e8d-4bfa-9cd6-32ac0593a343-kube-api-access-sbqj4" (OuterVolumeSpecName: "kube-api-access-sbqj4") pod "7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" (UID: "7ba8affc-3e8d-4bfa-9cd6-32ac0593a343"). InnerVolumeSpecName "kube-api-access-sbqj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:16.190868 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.184699    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqj4\" (UniqueName: \"kubernetes.io/projected/7ba8affc-3e8d-4bfa-9cd6-32ac0593a343-kube-api-access-sbqj4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:16.810193 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.810153    2579 generic.go:356] "Generic (PLEG): container finished" podID="7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" containerID="7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9" exitCode=137
Sep 12 02:59:16.810342 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.810205    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:16.810342 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.810259    2579 scope.go:117] "RemoveContainer" containerID="7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9"
Sep 12 02:59:16.819529 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.819510    2579 scope.go:117] "RemoveContainer" containerID="7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9"
Sep 12 02:59:16.819805 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:16.819786    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9\": container with ID starting with 7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9 not found: ID does not exist" containerID="7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9"
Sep 12 02:59:16.819860 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:16.819813    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9"} err="failed to get container status \"7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9\": rpc error: code = NotFound desc = could not find container \"7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9\": container with ID starting with 7eae9e4dfd1469e7f8a671d2ed13a3f3c8cc9a2a3235e6a66909d284f26d38e9 not found: ID does not exist"
Sep 12 02:59:16.820157 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:16.820132    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7879-6b7eb0\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" pod="opendatahub/test-curl-7879-6b7eb0"
Sep 12 02:59:17.250197 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:17.250159    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" path="/var/lib/kubelet/pods/7ba8affc-3e8d-4bfa-9cd6-32ac0593a343/volumes"
Sep 12 02:59:21.124972 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.124940    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"]
Sep 12 02:59:21.125554 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.125538    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" containerName="test-curl-7879-6b7eb0"
Sep 12 02:59:21.125600 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.125558    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" containerName="test-curl-7879-6b7eb0"
Sep 12 02:59:21.125701 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.125690    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba8affc-3e8d-4bfa-9cd6-32ac0593a343" containerName="test-curl-7879-6b7eb0"
Sep 12 02:59:21.130748 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.130724    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.133007 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.132911    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1c9060bc-serving-cert\""
Sep 12 02:59:21.139998 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.139973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"]
Sep 12 02:59:21.232196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.232166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-metrics-certs\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.232196 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.232197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00631628-4ba9-4020-953a-0ac13656178d-openshift-service-ca\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.232411 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.232221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.232411 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.232239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xprw\" (UniqueName: \"kubernetes.io/projected/00631628-4ba9-4020-953a-0ac13656178d-kube-api-access-6xprw\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.332723 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.332686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-metrics-certs\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.332723 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.332719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00631628-4ba9-4020-953a-0ac13656178d-openshift-service-ca\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.332963 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.332742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.332963 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.332762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xprw\" (UniqueName: \"kubernetes.io/projected/00631628-4ba9-4020-953a-0ac13656178d-kube-api-access-6xprw\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.332963 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:21.332899    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-serving-cert: secret "maas-api-e2e-mt-1c9060bc-serving-cert" not found
Sep 12 02:59:21.332963 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:21.332964    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls podName:00631628-4ba9-4020-953a-0ac13656178d nodeName:}" failed. No retries permitted until 2026-09-12 02:59:21.832946223 +0000 UTC m=+1401.100979850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls") pod "maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8" (UID: "00631628-4ba9-4020-953a-0ac13656178d") : secret "maas-api-e2e-mt-1c9060bc-serving-cert" not found
Sep 12 02:59:21.333466 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.333448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00631628-4ba9-4020-953a-0ac13656178d-openshift-service-ca\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.335394 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.335373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-metrics-certs\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.343738 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.343711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xprw\" (UniqueName: \"kubernetes.io/projected/00631628-4ba9-4020-953a-0ac13656178d-kube-api-access-6xprw\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.837691 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.837657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:21.840227 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:21.840201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls\") pod \"maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:22.044350 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:22.044318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:22.314641 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:22.314614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"]
Sep 12 02:59:22.316789 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:59:22.316754    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00631628_4ba9_4020_953a_0ac13656178d.slice/crio-a5d1825803b17b34c379eb55fa8b44d1fc00ca4376a14ba32d6c6004b1c9c300 WatchSource:0}: Error finding container a5d1825803b17b34c379eb55fa8b44d1fc00ca4376a14ba32d6c6004b1c9c300: Status 404 returned error can't find the container with id a5d1825803b17b34c379eb55fa8b44d1fc00ca4376a14ba32d6c6004b1c9c300
Sep 12 02:59:22.846517 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:22.846480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8" event={"ID":"00631628-4ba9-4020-953a-0ac13656178d","Type":"ContainerStarted","Data":"eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3"}
Sep 12 02:59:22.846696 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:22.846540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8" event={"ID":"00631628-4ba9-4020-953a-0ac13656178d","Type":"ContainerStarted","Data":"a5d1825803b17b34c379eb55fa8b44d1fc00ca4376a14ba32d6c6004b1c9c300"}
Sep 12 02:59:22.846770 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:22.846694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:22.864815 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:22.864763    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8" podStartSLOduration=1.443003757 podStartE2EDuration="1.86474957s" totalImagesPullingTime="421.745813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:22.862330959 +0000 UTC m=+1402.130364607" watchObservedRunningTime="2026-09-12 02:59:22.86474957 +0000 UTC m=+1402.132783217"
Sep 12 02:59:28.645331 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:28.645296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 02:59:28.859201 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:28.859175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 02:59:31.699853 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.699821    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"]
Sep 12 02:59:31.703593 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.703574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.705716 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.705694    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"e2e-conflict-a-40f128f2-kserve-self-signed-certs\""
Sep 12 02:59:31.711217 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.711196    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"]
Sep 12 02:59:31.835455 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.835424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-dshm\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.835631 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.835471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-tmp-dir\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.835631 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.835527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-model-cache\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.835631 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.835560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458b3a36-a8cc-45a3-811f-7372033fa4b5-tls-certs\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.835631 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.835624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-home\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.835795 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.835650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfgp2\" (UniqueName: \"kubernetes.io/projected/458b3a36-a8cc-45a3-811f-7372033fa4b5-kube-api-access-rfgp2\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936501 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-dshm\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936651 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-tmp-dir\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936651 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-model-cache\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936651 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458b3a36-a8cc-45a3-811f-7372033fa4b5-tls-certs\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936651 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-home\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936651 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfgp2\" (UniqueName: \"kubernetes.io/projected/458b3a36-a8cc-45a3-811f-7372033fa4b5-kube-api-access-rfgp2\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-tmp-dir\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.936989 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.936949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-model-cache\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.937075 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.937035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-home\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.938891 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.938861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-dshm\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.939153 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.939135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458b3a36-a8cc-45a3-811f-7372033fa4b5-tls-certs\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:31.943499 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:31.943479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfgp2\" (UniqueName: \"kubernetes.io/projected/458b3a36-a8cc-45a3-811f-7372033fa4b5-kube-api-access-rfgp2\") pod \"e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:32.015010 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:32.014948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:32.287311 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:32.287289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"]
Sep 12 02:59:32.289755 ip-10-0-130-57 kubenswrapper[2579]: W0912 02:59:32.289729    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod458b3a36_a8cc_45a3_811f_7372033fa4b5.slice/crio-40a54a1c479d9e9e3525b0dcf2191d003609ae38ef7b4b5d7d6647b7e2e1f23b WatchSource:0}: Error finding container 40a54a1c479d9e9e3525b0dcf2191d003609ae38ef7b4b5d7d6647b7e2e1f23b: Status 404 returned error can't find the container with id 40a54a1c479d9e9e3525b0dcf2191d003609ae38ef7b4b5d7d6647b7e2e1f23b
Sep 12 02:59:32.903189 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:32.903154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc" event={"ID":"458b3a36-a8cc-45a3-811f-7372033fa4b5","Type":"ContainerStarted","Data":"93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534"}
Sep 12 02:59:32.903189 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:32.903195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc" event={"ID":"458b3a36-a8cc-45a3-811f-7372033fa4b5","Type":"ContainerStarted","Data":"40a54a1c479d9e9e3525b0dcf2191d003609ae38ef7b4b5d7d6647b7e2e1f23b"}
Sep 12 02:59:32.926878 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:32.926830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc" podStartSLOduration=1.926813406 podStartE2EDuration="1.926813406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:32.924496844 +0000 UTC m=+1412.192530507" watchObservedRunningTime="2026-09-12 02:59:32.926813406 +0000 UTC m=+1412.194847052"
Sep 12 02:59:38.256987 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:38.256957    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"]
Sep 12 02:59:42.015998 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:42.015965    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:42.016424 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:42.016011    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:42.027972 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:42.027950    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:42.400190 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:42.400072    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"]
Sep 12 02:59:42.970654 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:42.970625    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 02:59:53.277045 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.276942    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n" podUID="5b619077-8fa2-4c8f-9115-cd0c5785a4d3" containerName="main" containerID="cri-o://8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53" gracePeriod=45
Sep 12 02:59:53.532900 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.532836    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:59:53.642901 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.642869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-model-cache\") pod \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") "
Sep 12 02:59:53.643087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.642929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-dshm\") pod \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") "
Sep 12 02:59:53.643087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-home\") pod \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") "
Sep 12 02:59:53.643087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643026    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sczc4\" (UniqueName: \"kubernetes.io/projected/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-kube-api-access-sczc4\") pod \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") "
Sep 12 02:59:53.643087 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643082    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tmp-dir\") pod \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") "
Sep 12 02:59:53.643355 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643165    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b619077-8fa2-4c8f-9115-cd0c5785a4d3" (UID: "5b619077-8fa2-4c8f-9115-cd0c5785a4d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:53.643355 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643215    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tls-certs\") pod \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\" (UID: \"5b619077-8fa2-4c8f-9115-cd0c5785a4d3\") "
Sep 12 02:59:53.643355 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-home" (OuterVolumeSpecName: "home") pod "5b619077-8fa2-4c8f-9115-cd0c5785a4d3" (UID: "5b619077-8fa2-4c8f-9115-cd0c5785a4d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:53.643473 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b619077-8fa2-4c8f-9115-cd0c5785a4d3" (UID: "5b619077-8fa2-4c8f-9115-cd0c5785a4d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:53.643636 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643619    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:53.643697 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643640    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:53.643697 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.643648    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:53.645383 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.645354    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-kube-api-access-sczc4" (OuterVolumeSpecName: "kube-api-access-sczc4") pod "5b619077-8fa2-4c8f-9115-cd0c5785a4d3" (UID: "5b619077-8fa2-4c8f-9115-cd0c5785a4d3"). InnerVolumeSpecName "kube-api-access-sczc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:53.645383 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.645369    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b619077-8fa2-4c8f-9115-cd0c5785a4d3" (UID: "5b619077-8fa2-4c8f-9115-cd0c5785a4d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:59:53.645383 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.645378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-dshm" (OuterVolumeSpecName: "dshm") pod "5b619077-8fa2-4c8f-9115-cd0c5785a4d3" (UID: "5b619077-8fa2-4c8f-9115-cd0c5785a4d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:53.745009 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.744982    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:53.745009 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.745004    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:53.745009 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:53.745013    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczc4\" (UniqueName: \"kubernetes.io/projected/5b619077-8fa2-4c8f-9115-cd0c5785a4d3-kube-api-access-sczc4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:54.019544 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.019507    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b619077-8fa2-4c8f-9115-cd0c5785a4d3" containerID="8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53" exitCode=0
Sep 12 02:59:54.019734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.019572    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"
Sep 12 02:59:54.019734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.019604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n" event={"ID":"5b619077-8fa2-4c8f-9115-cd0c5785a4d3","Type":"ContainerDied","Data":"8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53"}
Sep 12 02:59:54.019734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.019661    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n" event={"ID":"5b619077-8fa2-4c8f-9115-cd0c5785a4d3","Type":"ContainerDied","Data":"6384ea6e749d80ca943fe88ac999f1d32dc854171822311378be40c90095350f"}
Sep 12 02:59:54.019734 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.019684    2579 scope.go:117] "RemoveContainer" containerID="8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53"
Sep 12 02:59:54.029957 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.029940    2579 scope.go:117] "RemoveContainer" containerID="8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53"
Sep 12 02:59:54.030321 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:54.030298    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53\": container with ID starting with 8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53 not found: ID does not exist" containerID="8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53"
Sep 12 02:59:54.030407 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.030333    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53"} err="failed to get container status \"8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53\": rpc error: code = NotFound desc = could not find container \"8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53\": container with ID starting with 8430a116e98d5230432cf607cd6d54d8ff1f2439e19a57e4e3cb037c1e4bbc53 not found: ID does not exist"
Sep 12 02:59:54.040691 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.040669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"]
Sep 12 02:59:54.045861 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:54.045841    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a07b10/test-model-a07b10-kserve-7f4bf5d7d7-65x7n"]
Sep 12 02:59:55.249709 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:55.249668    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b619077-8fa2-4c8f-9115-cd0c5785a4d3" path="/var/lib/kubelet/pods/5b619077-8fa2-4c8f-9115-cd0c5785a4d3/volumes"
Sep 12 02:59:57.422176 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.422124    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq" podUID="9473f2db-432a-445c-a46a-d6bd5680205b" containerName="main" containerID="cri-o://c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716" gracePeriod=45
Sep 12 02:59:57.680394 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.680372    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:59:57.781925 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.781894    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-home\") pod \"9473f2db-432a-445c-a46a-d6bd5680205b\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") "
Sep 12 02:59:57.782117 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.781976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-tmp-dir\") pod \"9473f2db-432a-445c-a46a-d6bd5680205b\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") "
Sep 12 02:59:57.782117 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782006    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdv66\" (UniqueName: \"kubernetes.io/projected/9473f2db-432a-445c-a46a-d6bd5680205b-kube-api-access-hdv66\") pod \"9473f2db-432a-445c-a46a-d6bd5680205b\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") "
Sep 12 02:59:57.782250 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782122    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-model-cache\") pod \"9473f2db-432a-445c-a46a-d6bd5680205b\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") "
Sep 12 02:59:57.782250 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9473f2db-432a-445c-a46a-d6bd5680205b-tls-certs\") pod \"9473f2db-432a-445c-a46a-d6bd5680205b\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") "
Sep 12 02:59:57.782250 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782200    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-home" (OuterVolumeSpecName: "home") pod "9473f2db-432a-445c-a46a-d6bd5680205b" (UID: "9473f2db-432a-445c-a46a-d6bd5680205b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:57.782250 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9473f2db-432a-445c-a46a-d6bd5680205b" (UID: "9473f2db-432a-445c-a46a-d6bd5680205b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:57.782250 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-dshm\") pod \"9473f2db-432a-445c-a46a-d6bd5680205b\" (UID: \"9473f2db-432a-445c-a46a-d6bd5680205b\") "
Sep 12 02:59:57.782515 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782348    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9473f2db-432a-445c-a46a-d6bd5680205b" (UID: "9473f2db-432a-445c-a46a-d6bd5680205b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:57.782595 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782575    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:57.782638 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782624    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:57.782677 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.782639    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:57.784404 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.784382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-dshm" (OuterVolumeSpecName: "dshm") pod "9473f2db-432a-445c-a46a-d6bd5680205b" (UID: "9473f2db-432a-445c-a46a-d6bd5680205b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:59:57.784509 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.784411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9473f2db-432a-445c-a46a-d6bd5680205b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9473f2db-432a-445c-a46a-d6bd5680205b" (UID: "9473f2db-432a-445c-a46a-d6bd5680205b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:59:57.784509 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.784459    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9473f2db-432a-445c-a46a-d6bd5680205b-kube-api-access-hdv66" (OuterVolumeSpecName: "kube-api-access-hdv66") pod "9473f2db-432a-445c-a46a-d6bd5680205b" (UID: "9473f2db-432a-445c-a46a-d6bd5680205b"). InnerVolumeSpecName "kube-api-access-hdv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:57.883339 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.883305    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9473f2db-432a-445c-a46a-d6bd5680205b-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:57.883339 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.883336    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9473f2db-432a-445c-a46a-d6bd5680205b-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:57.883339 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:57.883345    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdv66\" (UniqueName: \"kubernetes.io/projected/9473f2db-432a-445c-a46a-d6bd5680205b-kube-api-access-hdv66\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 02:59:58.042156 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.042051    2579 generic.go:356] "Generic (PLEG): container finished" podID="9473f2db-432a-445c-a46a-d6bd5680205b" containerID="c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716" exitCode=0
Sep 12 02:59:58.042156 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.042110    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"
Sep 12 02:59:58.042363 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.042144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq" event={"ID":"9473f2db-432a-445c-a46a-d6bd5680205b","Type":"ContainerDied","Data":"c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716"}
Sep 12 02:59:58.042363 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.042198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq" event={"ID":"9473f2db-432a-445c-a46a-d6bd5680205b","Type":"ContainerDied","Data":"2c2650bfad12b8b7e6e3b3afd650510b7c01e79499d65bc6c20aff8cd1d258ad"}
Sep 12 02:59:58.042363 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.042224    2579 scope.go:117] "RemoveContainer" containerID="c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716"
Sep 12 02:59:58.051649 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.051635    2579 scope.go:117] "RemoveContainer" containerID="c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716"
Sep 12 02:59:58.051889 ip-10-0-130-57 kubenswrapper[2579]: E0912 02:59:58.051874    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716\": container with ID starting with c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716 not found: ID does not exist" containerID="c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716"
Sep 12 02:59:58.051942 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.051896    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716"} err="failed to get container status \"c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716\": rpc error: code = NotFound desc = could not find container \"c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716\": container with ID starting with c2dd48ae333c2eede708b380d22976a1c1a36b22b8b03af15fe6b3a37e88a716 not found: ID does not exist"
Sep 12 02:59:58.062324 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.062305    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"]
Sep 12 02:59:58.067675 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:58.067655    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f93121/test-model-f93121-kserve-f58f4cccc-fgczq"]
Sep 12 02:59:59.249683 ip-10-0-130-57 kubenswrapper[2579]: I0912 02:59:59.249654    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9473f2db-432a-445c-a46a-d6bd5680205b" path="/var/lib/kubelet/pods/9473f2db-432a-445c-a46a-d6bd5680205b/volumes"
Sep 12 03:00:00.155715 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.155683    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"]
Sep 12 03:00:00.156344 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.156327    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b619077-8fa2-4c8f-9115-cd0c5785a4d3" containerName="main"
Sep 12 03:00:00.156392 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.156349    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b619077-8fa2-4c8f-9115-cd0c5785a4d3" containerName="main"
Sep 12 03:00:00.156392 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.156368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9473f2db-432a-445c-a46a-d6bd5680205b" containerName="main"
Sep 12 03:00:00.156392 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.156376    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9473f2db-432a-445c-a46a-d6bd5680205b" containerName="main"
Sep 12 03:00:00.156514 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.156502    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9473f2db-432a-445c-a46a-d6bd5680205b" containerName="main"
Sep 12 03:00:00.156551 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.156521    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b619077-8fa2-4c8f-9115-cd0c5785a4d3" containerName="main"
Sep 12 03:00:00.159981 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.159960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"
Sep 12 03:00:00.162335 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.162314    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4g6hp\""
Sep 12 03:00:00.167268 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.167247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"]
Sep 12 03:00:00.172626 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.172611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"
Sep 12 03:00:00.443929 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:00.443906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"]
Sep 12 03:00:00.446277 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:00:00.446234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b881678_7bd6_49bf_b67d_92f4fe9dab9c.slice/crio-81ea93940fdaf5006a08b448f883c0ab27dc596a9c330c440734d01537d7dc21 WatchSource:0}: Error finding container 81ea93940fdaf5006a08b448f883c0ab27dc596a9c330c440734d01537d7dc21: Status 404 returned error can't find the container with id 81ea93940fdaf5006a08b448f883c0ab27dc596a9c330c440734d01537d7dc21
Sep 12 03:00:01.061066 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:01.061027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk" event={"ID":"8b881678-7bd6-49bf-b67d-92f4fe9dab9c","Type":"ContainerStarted","Data":"81ea93940fdaf5006a08b448f883c0ab27dc596a9c330c440734d01537d7dc21"}
Sep 12 03:00:04.080354 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:04.080321    2579 generic.go:356] "Generic (PLEG): container finished" podID="8b881678-7bd6-49bf-b67d-92f4fe9dab9c" containerID="e2a6861dcf2ccf65b27353da2083256561c819e37e2133753cbaa190db9d723f" exitCode=0
Sep 12 03:00:04.080688 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:04.080445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk" event={"ID":"8b881678-7bd6-49bf-b67d-92f4fe9dab9c","Type":"ContainerDied","Data":"e2a6861dcf2ccf65b27353da2083256561c819e37e2133753cbaa190db9d723f"}
Sep 12 03:00:05.224065 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:05.224044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"
Sep 12 03:00:05.258340 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:05.258314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:00:06.094548 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:06.094516    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"
Sep 12 03:00:06.094732 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:06.094518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk" event={"ID":"8b881678-7bd6-49bf-b67d-92f4fe9dab9c","Type":"ContainerDied","Data":"81ea93940fdaf5006a08b448f883c0ab27dc596a9c330c440734d01537d7dc21"}
Sep 12 03:00:06.094732 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:06.094667    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81ea93940fdaf5006a08b448f883c0ab27dc596a9c330c440734d01537d7dc21"
Sep 12 03:00:17.013765 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.013723    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"]
Sep 12 03:00:17.014416 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.014393    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b881678-7bd6-49bf-b67d-92f4fe9dab9c" containerName="cleanup"
Sep 12 03:00:17.014416 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.014417    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b881678-7bd6-49bf-b67d-92f4fe9dab9c" containerName="cleanup"
Sep 12 03:00:17.014603 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.014549    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b881678-7bd6-49bf-b67d-92f4fe9dab9c" containerName="cleanup"
Sep 12 03:00:17.017890 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.017870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.020219 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.020194    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1c9060bc\"/\"kube-root-ca.crt\""
Sep 12 03:00:17.021209 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.020979    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1c9060bc\"/\"openshift-service-ca.crt\""
Sep 12 03:00:17.021209 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.021017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1c9060bc\"/\"default-dockercfg-p9gxd\""
Sep 12 03:00:17.021209 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.021019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1c9060bc\"/\"e2e-lifecycle-model-1c9060bc-kserve-self-signed-certs\""
Sep 12 03:00:17.027292 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.027274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"]
Sep 12 03:00:17.176006 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.175976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79j9m\" (UniqueName: \"kubernetes.io/projected/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-kube-api-access-79j9m\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.176206 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.176037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-model-cache\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.176206 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.176081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-home\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.176206 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.176175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tmp-dir\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.176334 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.176211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tls-certs\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.176334 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.176229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-dshm\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277139 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-model-cache\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277139 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-home\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277383 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tmp-dir\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277383 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tls-certs\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277383 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-dshm\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277383 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79j9m\" (UniqueName: \"kubernetes.io/projected/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-kube-api-access-79j9m\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277580 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-model-cache\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277580 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tmp-dir\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.277645 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.277575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-home\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.279587 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.279559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-dshm\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.279892 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.279875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tls-certs\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.284270 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.284248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79j9m\" (UniqueName: \"kubernetes.io/projected/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-kube-api-access-79j9m\") pod \"e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") " pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.328909 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.328886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:17.607438 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:17.607408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"]
Sep 12 03:00:17.608839 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:00:17.608812    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddab45b64_ca8f_4b37_8dfe_7eb812836c1b.slice/crio-1bdb2d174014354949eda36cc449ac003189272379e44a7020f5d34d1a893ee5 WatchSource:0}: Error finding container 1bdb2d174014354949eda36cc449ac003189272379e44a7020f5d34d1a893ee5: Status 404 returned error can't find the container with id 1bdb2d174014354949eda36cc449ac003189272379e44a7020f5d34d1a893ee5
Sep 12 03:00:18.158689 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:18.158651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5" event={"ID":"dab45b64-ca8f-4b37-8dfe-7eb812836c1b","Type":"ContainerStarted","Data":"46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c"}
Sep 12 03:00:18.158689 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:18.158690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5" event={"ID":"dab45b64-ca8f-4b37-8dfe-7eb812836c1b","Type":"ContainerStarted","Data":"1bdb2d174014354949eda36cc449ac003189272379e44a7020f5d34d1a893ee5"}
Sep 12 03:00:18.182199 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:18.176249    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5" podStartSLOduration=2.176227385 podStartE2EDuration="2.176227385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:00:18.174555801 +0000 UTC m=+1457.442589456" watchObservedRunningTime="2026-09-12 03:00:18.176227385 +0000 UTC m=+1457.444261033"
Sep 12 03:00:25.925910 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:25.925874    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"]
Sep 12 03:00:25.926725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:25.926175    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w" podUID="bb4d163d-b8cc-49db-93d9-981928dfcea2" containerName="maas-api" containerID="cri-o://fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca" gracePeriod=30
Sep 12 03:00:26.180728 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.180704    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 03:00:26.204750 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.204713    2579 generic.go:356] "Generic (PLEG): container finished" podID="bb4d163d-b8cc-49db-93d9-981928dfcea2" containerID="fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca" exitCode=0
Sep 12 03:00:26.204952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.204763    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"
Sep 12 03:00:26.204952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.204808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w" event={"ID":"bb4d163d-b8cc-49db-93d9-981928dfcea2","Type":"ContainerDied","Data":"fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca"}
Sep 12 03:00:26.204952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.204864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w" event={"ID":"bb4d163d-b8cc-49db-93d9-981928dfcea2","Type":"ContainerDied","Data":"bbee99dba2412446804d9649b62ff74d978bdaed27307a4016aaea47cd5c9c70"}
Sep 12 03:00:26.204952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.204896    2579 scope.go:117] "RemoveContainer" containerID="fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca"
Sep 12 03:00:26.218303 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.218270    2579 scope.go:117] "RemoveContainer" containerID="fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca"
Sep 12 03:00:26.218620 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:00:26.218598    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca\": container with ID starting with fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca not found: ID does not exist" containerID="fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca"
Sep 12 03:00:26.218713 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.218629    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca"} err="failed to get container status \"fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca\": rpc error: code = NotFound desc = could not find container \"fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca\": container with ID starting with fdc3a3e16831133e4855966f0ff2eda4a85c838d8da57bf194552027c99813ca not found: ID does not exist"
Sep 12 03:00:26.256053 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.256019    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-metrics-certs\") pod \"bb4d163d-b8cc-49db-93d9-981928dfcea2\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") "
Sep 12 03:00:26.256180 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.256148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4d163d-b8cc-49db-93d9-981928dfcea2-openshift-service-ca\") pod \"bb4d163d-b8cc-49db-93d9-981928dfcea2\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") "
Sep 12 03:00:26.256180 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.256179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68llr\" (UniqueName: \"kubernetes.io/projected/bb4d163d-b8cc-49db-93d9-981928dfcea2-kube-api-access-68llr\") pod \"bb4d163d-b8cc-49db-93d9-981928dfcea2\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") "
Sep 12 03:00:26.256299 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.256228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls\") pod \"bb4d163d-b8cc-49db-93d9-981928dfcea2\" (UID: \"bb4d163d-b8cc-49db-93d9-981928dfcea2\") "
Sep 12 03:00:26.256466 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.256438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4d163d-b8cc-49db-93d9-981928dfcea2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb4d163d-b8cc-49db-93d9-981928dfcea2" (UID: "bb4d163d-b8cc-49db-93d9-981928dfcea2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:00:26.256571 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.256541    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4d163d-b8cc-49db-93d9-981928dfcea2-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.258381 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.258355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4d163d-b8cc-49db-93d9-981928dfcea2-kube-api-access-68llr" (OuterVolumeSpecName: "kube-api-access-68llr") pod "bb4d163d-b8cc-49db-93d9-981928dfcea2" (UID: "bb4d163d-b8cc-49db-93d9-981928dfcea2"). InnerVolumeSpecName "kube-api-access-68llr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:00:26.258499 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.258378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb4d163d-b8cc-49db-93d9-981928dfcea2" (UID: "bb4d163d-b8cc-49db-93d9-981928dfcea2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:00:26.258499 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.258419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb4d163d-b8cc-49db-93d9-981928dfcea2" (UID: "bb4d163d-b8cc-49db-93d9-981928dfcea2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:00:26.357286 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.357245    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.357286 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.357280    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4d163d-b8cc-49db-93d9-981928dfcea2-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.357286 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.357290    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68llr\" (UniqueName: \"kubernetes.io/projected/bb4d163d-b8cc-49db-93d9-981928dfcea2-kube-api-access-68llr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.526773 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.526738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"]
Sep 12 03:00:26.531835 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:26.531808    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f93121-86998b4774-k2d7w"]
Sep 12 03:00:27.249749 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:27.249718    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb4d163d-b8cc-49db-93d9-981928dfcea2" path="/var/lib/kubelet/pods/bb4d163d-b8cc-49db-93d9-981928dfcea2/volumes"
Sep 12 03:00:27.250140 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:27.249985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:00:27.329189 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:27.329163    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:27.329330 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:27.329221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:27.341645 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:27.341626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:28.227916 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:28.227888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:00:40.008837 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.008803    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"]
Sep 12 03:00:40.009466 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.009446    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb4d163d-b8cc-49db-93d9-981928dfcea2" containerName="maas-api"
Sep 12 03:00:40.009466 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.009466    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb4d163d-b8cc-49db-93d9-981928dfcea2" containerName="maas-api"
Sep 12 03:00:40.009620 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.009564    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb4d163d-b8cc-49db-93d9-981928dfcea2" containerName="maas-api"
Sep 12 03:00:40.017315 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.017291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.019748 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.019708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f91169-serving-cert\""
Sep 12 03:00:40.021386 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.021363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"]
Sep 12 03:00:40.087673 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.087645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tv6\" (UniqueName: \"kubernetes.io/projected/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-kube-api-access-h9tv6\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.087835 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.087682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.087835 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.087706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.087924 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.087857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.188798 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.188757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.188963 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.188905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.188963 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.188952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tv6\" (UniqueName: \"kubernetes.io/projected/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-kube-api-access-h9tv6\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.189227 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.189052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.189321 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:00:40.189194    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-serving-cert: secret "maas-api-e2e-shared-a-w3-f91169-serving-cert" not found
Sep 12 03:00:40.189419 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:00:40.189389    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls podName:c7d44a7f-8e02-4fff-89d6-0204b10fb1c4 nodeName:}" failed. No retries permitted until 2026-09-12 03:00:40.689358267 +0000 UTC m=+1479.957391910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls") pod "maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr" (UID: "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4") : secret "maas-api-e2e-shared-a-w3-f91169-serving-cert" not found
Sep 12 03:00:40.189612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.189569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.192109 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.192073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.199279 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.199254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tv6\" (UniqueName: \"kubernetes.io/projected/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-kube-api-access-h9tv6\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.696045 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.696000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.698901 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.698876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:40.931335 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:40.931301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:41.203079 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:41.203048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"]
Sep 12 03:00:41.206078 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:00:41.206052    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d44a7f_8e02_4fff_89d6_0204b10fb1c4.slice/crio-03c3ba537b5091761c21603670e40d4c2772d9715eda2db71fbb71a163e1682a WatchSource:0}: Error finding container 03c3ba537b5091761c21603670e40d4c2772d9715eda2db71fbb71a163e1682a: Status 404 returned error can't find the container with id 03c3ba537b5091761c21603670e40d4c2772d9715eda2db71fbb71a163e1682a
Sep 12 03:00:41.284845 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:41.284776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr" event={"ID":"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4","Type":"ContainerStarted","Data":"03c3ba537b5091761c21603670e40d4c2772d9715eda2db71fbb71a163e1682a"}
Sep 12 03:00:42.292143 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:42.292072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr" event={"ID":"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4","Type":"ContainerStarted","Data":"530fb1877cae96018cbbb1843ee227979de874825dea4a4fbe581acea2a4328f"}
Sep 12 03:00:42.292534 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:42.292175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:42.311523 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:42.311479    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr" podStartSLOduration=2.790688283 podStartE2EDuration="3.311465043s" totalImagesPullingTime="520.77676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:00:42.310603593 +0000 UTC m=+1481.578637267" watchObservedRunningTime="2026-09-12 03:00:42.311465043 +0000 UTC m=+1481.579498690"
Sep 12 03:00:48.303203 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:48.303166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:00:58.354114 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:00:58.354066    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:01:14.861069 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.861035    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"]
Sep 12 03:01:14.866544 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.866524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:14.868641 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.868619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e401bd-serving-cert\""
Sep 12 03:01:14.875927 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.875869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"]
Sep 12 03:01:14.932760 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.932730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:14.932909 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.932813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:14.932909 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.932839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kkc\" (UniqueName: \"kubernetes.io/projected/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-kube-api-access-68kkc\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:14.932988 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:14.932920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.033764 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.033732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.033928 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.033813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.033928 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.033869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.033928 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.033912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kkc\" (UniqueName: \"kubernetes.io/projected/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-kube-api-access-68kkc\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.034501 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.034478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.036441 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.036420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.036523 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.036455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.041280 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.041254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kkc\" (UniqueName: \"kubernetes.io/projected/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-kube-api-access-68kkc\") pod \"maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.181040 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.181020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:15.459239 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:15.459149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"]
Sep 12 03:01:15.467479 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:01:15.467450    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1c44b5_6d09_4cf0_ab3f_e83b5975f54e.slice/crio-61cfb1be88eaeba77f8a0a025ca58aa9a7f3cc360e4dc467196a5d38a05b41f3 WatchSource:0}: Error finding container 61cfb1be88eaeba77f8a0a025ca58aa9a7f3cc360e4dc467196a5d38a05b41f3: Status 404 returned error can't find the container with id 61cfb1be88eaeba77f8a0a025ca58aa9a7f3cc360e4dc467196a5d38a05b41f3
Sep 12 03:01:16.475632 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:16.475594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb" event={"ID":"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e","Type":"ContainerStarted","Data":"5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807"}
Sep 12 03:01:16.476064 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:16.475648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb" event={"ID":"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e","Type":"ContainerStarted","Data":"61cfb1be88eaeba77f8a0a025ca58aa9a7f3cc360e4dc467196a5d38a05b41f3"}
Sep 12 03:01:16.476064 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:16.475744    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:16.491667 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:16.491625    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb" podStartSLOduration=2.121440161 podStartE2EDuration="2.491612814s" totalImagesPullingTime="370.172653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:01:16.489411872 +0000 UTC m=+1515.757445521" watchObservedRunningTime="2026-09-12 03:01:16.491612814 +0000 UTC m=+1515.759646460"
Sep 12 03:01:22.486320 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:22.486247    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:01:33.110251 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.110219    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"]
Sep 12 03:01:33.113869 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.113849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.117231 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.117210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"openshift-service-ca.crt\""
Sep 12 03:01:33.117356 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.117241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"default-dockercfg-pc6s2\""
Sep 12 03:01:33.117356 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.117211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"auth-test-model-f91169-kserve-self-signed-certs\""
Sep 12 03:01:33.117496 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.117479    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"kube-root-ca.crt\""
Sep 12 03:01:33.122160 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.122137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"]
Sep 12 03:01:33.213458 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.213432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpng6\" (UniqueName: \"kubernetes.io/projected/c35849b3-dedd-4637-9c1c-fb90700198bb-kube-api-access-zpng6\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.213605 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.213464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35849b3-dedd-4637-9c1c-fb90700198bb-tls-certs\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.213605 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.213482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-model-cache\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.213605 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.213530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-tmp-dir\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.213605 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.213585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-home\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.213751 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.213635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-dshm\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.252448 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.252419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:01:33.314218 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpng6\" (UniqueName: \"kubernetes.io/projected/c35849b3-dedd-4637-9c1c-fb90700198bb-kube-api-access-zpng6\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.314218 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35849b3-dedd-4637-9c1c-fb90700198bb-tls-certs\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.314486 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-model-cache\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.314486 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-tmp-dir\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.314486 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-home\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.314767 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-model-cache\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.314855 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-tmp-dir\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.315015 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.314882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-home\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.315171 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.315152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-dshm\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.316971 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.316944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35849b3-dedd-4637-9c1c-fb90700198bb-tls-certs\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.317303 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.317279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-dshm\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.322126 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.322067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpng6\" (UniqueName: \"kubernetes.io/projected/c35849b3-dedd-4637-9c1c-fb90700198bb-kube-api-access-zpng6\") pod \"auth-test-model-f91169-kserve-554975595-dsxm8\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") " pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.426746 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.426680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:33.698049 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:33.698018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"]
Sep 12 03:01:33.700080 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:01:33.700040    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc35849b3_dedd_4637_9c1c_fb90700198bb.slice/crio-ea96a362309c93946aa20609cc2298a6ccb114a8a2c4a972db5e09679192b5fc WatchSource:0}: Error finding container ea96a362309c93946aa20609cc2298a6ccb114a8a2c4a972db5e09679192b5fc: Status 404 returned error can't find the container with id ea96a362309c93946aa20609cc2298a6ccb114a8a2c4a972db5e09679192b5fc
Sep 12 03:01:34.574879 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:34.574848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8" event={"ID":"c35849b3-dedd-4637-9c1c-fb90700198bb","Type":"ContainerStarted","Data":"c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf"}
Sep 12 03:01:34.575259 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:34.574885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8" event={"ID":"c35849b3-dedd-4637-9c1c-fb90700198bb","Type":"ContainerStarted","Data":"ea96a362309c93946aa20609cc2298a6ccb114a8a2c4a972db5e09679192b5fc"}
Sep 12 03:01:34.590467 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:34.590406    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8" podStartSLOduration=1.5903865339999999 podStartE2EDuration="1.590386534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:01:34.589569354 +0000 UTC m=+1533.857603023" watchObservedRunningTime="2026-09-12 03:01:34.590386534 +0000 UTC m=+1533.858420182"
Sep 12 03:01:35.564411 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.564383    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"]
Sep 12 03:01:35.564607 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.564582    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8" podUID="00631628-4ba9-4020-953a-0ac13656178d" containerName="maas-api" containerID="cri-o://eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3" gracePeriod=30
Sep 12 03:01:35.825179 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.825121    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 03:01:35.838385 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.838362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-metrics-certs\") pod \"00631628-4ba9-4020-953a-0ac13656178d\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") "
Sep 12 03:01:35.838520 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.838407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls\") pod \"00631628-4ba9-4020-953a-0ac13656178d\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") "
Sep 12 03:01:35.838520 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.838500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00631628-4ba9-4020-953a-0ac13656178d-openshift-service-ca\") pod \"00631628-4ba9-4020-953a-0ac13656178d\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") "
Sep 12 03:01:35.838638 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.838525    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xprw\" (UniqueName: \"kubernetes.io/projected/00631628-4ba9-4020-953a-0ac13656178d-kube-api-access-6xprw\") pod \"00631628-4ba9-4020-953a-0ac13656178d\" (UID: \"00631628-4ba9-4020-953a-0ac13656178d\") "
Sep 12 03:01:35.838862 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.838841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00631628-4ba9-4020-953a-0ac13656178d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00631628-4ba9-4020-953a-0ac13656178d" (UID: "00631628-4ba9-4020-953a-0ac13656178d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:01:35.840730 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.840698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00631628-4ba9-4020-953a-0ac13656178d" (UID: "00631628-4ba9-4020-953a-0ac13656178d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:01:35.840977 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.840926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "00631628-4ba9-4020-953a-0ac13656178d" (UID: "00631628-4ba9-4020-953a-0ac13656178d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:01:35.841228 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.841185    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00631628-4ba9-4020-953a-0ac13656178d-kube-api-access-6xprw" (OuterVolumeSpecName: "kube-api-access-6xprw") pod "00631628-4ba9-4020-953a-0ac13656178d" (UID: "00631628-4ba9-4020-953a-0ac13656178d"). InnerVolumeSpecName "kube-api-access-6xprw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:01:35.939612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.939582    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00631628-4ba9-4020-953a-0ac13656178d-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:35.939612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.939609    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xprw\" (UniqueName: \"kubernetes.io/projected/00631628-4ba9-4020-953a-0ac13656178d-kube-api-access-6xprw\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:35.939612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.939618    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:35.939809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:35.939627    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00631628-4ba9-4020-953a-0ac13656178d-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:36.589304 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.589274    2579 generic.go:356] "Generic (PLEG): container finished" podID="00631628-4ba9-4020-953a-0ac13656178d" containerID="eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3" exitCode=0
Sep 12 03:01:36.589502 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.589318    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"
Sep 12 03:01:36.589502 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.589331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8" event={"ID":"00631628-4ba9-4020-953a-0ac13656178d","Type":"ContainerDied","Data":"eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3"}
Sep 12 03:01:36.589502 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.589376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8" event={"ID":"00631628-4ba9-4020-953a-0ac13656178d","Type":"ContainerDied","Data":"a5d1825803b17b34c379eb55fa8b44d1fc00ca4376a14ba32d6c6004b1c9c300"}
Sep 12 03:01:36.589502 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.589401    2579 scope.go:117] "RemoveContainer" containerID="eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3"
Sep 12 03:01:36.599959 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.599942    2579 scope.go:117] "RemoveContainer" containerID="eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3"
Sep 12 03:01:36.600314 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:01:36.600292    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3\": container with ID starting with eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3 not found: ID does not exist" containerID="eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3"
Sep 12 03:01:36.600443 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.600321    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3"} err="failed to get container status \"eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3\": rpc error: code = NotFound desc = could not find container \"eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3\": container with ID starting with eee131cc21a6af5d07cc36e1341d502e65853754c90f36c77cc2e3cbdf25bdd3 not found: ID does not exist"
Sep 12 03:01:36.610566 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.610544    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"]
Sep 12 03:01:36.613848 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:36.613829    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c9060bc-6bfb4f65bf-hdfh8"]
Sep 12 03:01:37.249199 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:37.249167    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00631628-4ba9-4020-953a-0ac13656178d" path="/var/lib/kubelet/pods/00631628-4ba9-4020-953a-0ac13656178d/volumes"
Sep 12 03:01:43.427760 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:43.427716    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:43.427760 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:43.427767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:43.440251 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:43.440224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:43.644673 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:43.644642    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:01:44.034921 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:44.034879    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"]
Sep 12 03:01:44.274848 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:44.274811    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"]
Sep 12 03:01:59.055273 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.055213    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5" podUID="dab45b64-ca8f-4b37-8dfe-7eb812836c1b" containerName="main" containerID="cri-o://46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c" gracePeriod=45
Sep 12 03:01:59.304320 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.304285    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc" podUID="458b3a36-a8cc-45a3-811f-7372033fa4b5" containerName="main" containerID="cri-o://93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534" gracePeriod=45
Sep 12 03:01:59.304959 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.304940    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:01:59.466233 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.466204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-dshm\") pod \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") "
Sep 12 03:01:59.466389 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.466247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79j9m\" (UniqueName: \"kubernetes.io/projected/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-kube-api-access-79j9m\") pod \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") "
Sep 12 03:01:59.466389 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.466276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-home\") pod \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") "
Sep 12 03:01:59.466389 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.466304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tmp-dir\") pod \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") "
Sep 12 03:01:59.466389 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.466373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-model-cache\") pod \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") "
Sep 12 03:01:59.466622 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.466412    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tls-certs\") pod \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\" (UID: \"dab45b64-ca8f-4b37-8dfe-7eb812836c1b\") "
Sep 12 03:01:59.467612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.466671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-home" (OuterVolumeSpecName: "home") pod "dab45b64-ca8f-4b37-8dfe-7eb812836c1b" (UID: "dab45b64-ca8f-4b37-8dfe-7eb812836c1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.467612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.467235    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.467612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.467364    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "dab45b64-ca8f-4b37-8dfe-7eb812836c1b" (UID: "dab45b64-ca8f-4b37-8dfe-7eb812836c1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.467612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.467478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dab45b64-ca8f-4b37-8dfe-7eb812836c1b" (UID: "dab45b64-ca8f-4b37-8dfe-7eb812836c1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.470538 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.470512    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-dshm" (OuterVolumeSpecName: "dshm") pod "dab45b64-ca8f-4b37-8dfe-7eb812836c1b" (UID: "dab45b64-ca8f-4b37-8dfe-7eb812836c1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.472523 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.472496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-kube-api-access-79j9m" (OuterVolumeSpecName: "kube-api-access-79j9m") pod "dab45b64-ca8f-4b37-8dfe-7eb812836c1b" (UID: "dab45b64-ca8f-4b37-8dfe-7eb812836c1b"). InnerVolumeSpecName "kube-api-access-79j9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:01:59.473178 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.473158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dab45b64-ca8f-4b37-8dfe-7eb812836c1b" (UID: "dab45b64-ca8f-4b37-8dfe-7eb812836c1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:01:59.547027 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.547002    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 03:01:59.568760 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.568696    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.568760 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.568727    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79j9m\" (UniqueName: \"kubernetes.io/projected/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-kube-api-access-79j9m\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.568760 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.568744    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.568760 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.568757    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.568980 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.568769    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab45b64-ca8f-4b37-8dfe-7eb812836c1b-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.669863 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.669826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-tmp-dir\") pod \"458b3a36-a8cc-45a3-811f-7372033fa4b5\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") "
Sep 12 03:01:59.669863 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.669866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-home\") pod \"458b3a36-a8cc-45a3-811f-7372033fa4b5\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") "
Sep 12 03:01:59.670144 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.669947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfgp2\" (UniqueName: \"kubernetes.io/projected/458b3a36-a8cc-45a3-811f-7372033fa4b5-kube-api-access-rfgp2\") pod \"458b3a36-a8cc-45a3-811f-7372033fa4b5\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") "
Sep 12 03:01:59.670144 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.669968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-model-cache\") pod \"458b3a36-a8cc-45a3-811f-7372033fa4b5\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") "
Sep 12 03:01:59.670144 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458b3a36-a8cc-45a3-811f-7372033fa4b5-tls-certs\") pod \"458b3a36-a8cc-45a3-811f-7372033fa4b5\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") "
Sep 12 03:01:59.670144 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-dshm\") pod \"458b3a36-a8cc-45a3-811f-7372033fa4b5\" (UID: \"458b3a36-a8cc-45a3-811f-7372033fa4b5\") "
Sep 12 03:01:59.670362 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "458b3a36-a8cc-45a3-811f-7372033fa4b5" (UID: "458b3a36-a8cc-45a3-811f-7372033fa4b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.670362 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670222    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-home" (OuterVolumeSpecName: "home") pod "458b3a36-a8cc-45a3-811f-7372033fa4b5" (UID: "458b3a36-a8cc-45a3-811f-7372033fa4b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.670362 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "458b3a36-a8cc-45a3-811f-7372033fa4b5" (UID: "458b3a36-a8cc-45a3-811f-7372033fa4b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.670518 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670479    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.670518 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670492    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.670518 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.670500    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.672450 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.672426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-dshm" (OuterVolumeSpecName: "dshm") pod "458b3a36-a8cc-45a3-811f-7372033fa4b5" (UID: "458b3a36-a8cc-45a3-811f-7372033fa4b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:59.672450 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.672441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458b3a36-a8cc-45a3-811f-7372033fa4b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "458b3a36-a8cc-45a3-811f-7372033fa4b5" (UID: "458b3a36-a8cc-45a3-811f-7372033fa4b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:01:59.672597 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.672446    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458b3a36-a8cc-45a3-811f-7372033fa4b5-kube-api-access-rfgp2" (OuterVolumeSpecName: "kube-api-access-rfgp2") pod "458b3a36-a8cc-45a3-811f-7372033fa4b5" (UID: "458b3a36-a8cc-45a3-811f-7372033fa4b5"). InnerVolumeSpecName "kube-api-access-rfgp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:01:59.719848 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.719821    2579 generic.go:356] "Generic (PLEG): container finished" podID="dab45b64-ca8f-4b37-8dfe-7eb812836c1b" containerID="46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c" exitCode=0
Sep 12 03:01:59.719976 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.719868    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"
Sep 12 03:01:59.719976 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.719911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5" event={"ID":"dab45b64-ca8f-4b37-8dfe-7eb812836c1b","Type":"ContainerDied","Data":"46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c"}
Sep 12 03:01:59.719976 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.719959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5" event={"ID":"dab45b64-ca8f-4b37-8dfe-7eb812836c1b","Type":"ContainerDied","Data":"1bdb2d174014354949eda36cc449ac003189272379e44a7020f5d34d1a893ee5"}
Sep 12 03:01:59.719976 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.719976    2579 scope.go:117] "RemoveContainer" containerID="46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c"
Sep 12 03:01:59.721610 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.721589    2579 generic.go:356] "Generic (PLEG): container finished" podID="458b3a36-a8cc-45a3-811f-7372033fa4b5" containerID="93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534" exitCode=0
Sep 12 03:01:59.721733 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.721623    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"
Sep 12 03:01:59.721733 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.721621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc" event={"ID":"458b3a36-a8cc-45a3-811f-7372033fa4b5","Type":"ContainerDied","Data":"93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534"}
Sep 12 03:01:59.721851 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.721732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc" event={"ID":"458b3a36-a8cc-45a3-811f-7372033fa4b5","Type":"ContainerDied","Data":"40a54a1c479d9e9e3525b0dcf2191d003609ae38ef7b4b5d7d6647b7e2e1f23b"}
Sep 12 03:01:59.730608 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.730589    2579 scope.go:117] "RemoveContainer" containerID="46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c"
Sep 12 03:01:59.730892 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:01:59.730874    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c\": container with ID starting with 46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c not found: ID does not exist" containerID="46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c"
Sep 12 03:01:59.730952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.730900    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c"} err="failed to get container status \"46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c\": rpc error: code = NotFound desc = could not find container \"46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c\": container with ID starting with 46450afa7eafb0db6f4ccfbafab1872eac996458da3eef614bcf29b88115982c not found: ID does not exist"
Sep 12 03:01:59.730952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.730917    2579 scope.go:117] "RemoveContainer" containerID="93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534"
Sep 12 03:01:59.739062 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.739046    2579 scope.go:117] "RemoveContainer" containerID="93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534"
Sep 12 03:01:59.739348 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:01:59.739333    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534\": container with ID starting with 93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534 not found: ID does not exist" containerID="93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534"
Sep 12 03:01:59.739402 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.739353    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534"} err="failed to get container status \"93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534\": rpc error: code = NotFound desc = could not find container \"93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534\": container with ID starting with 93b27d9ebc75c0ce7b16cb8cda67ce2bef3a84768cfc56ca4c610e58e1cad534 not found: ID does not exist"
Sep 12 03:01:59.744620 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.744602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"]
Sep 12 03:01:59.746620 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.746599    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1c9060bc/e2e-lifecycle-model-1c9060bc-kserve-bdf46df9f-7vsz5"]
Sep 12 03:01:59.756877 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.756845    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"]
Sep 12 03:01:59.758581 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.758564    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-a-40f128f2-kserve-7cc8f44556-hn8kc"]
Sep 12 03:01:59.771244 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.771227    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458b3a36-a8cc-45a3-811f-7372033fa4b5-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.771244 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.771245    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458b3a36-a8cc-45a3-811f-7372033fa4b5-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:01:59.771349 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:01:59.771254    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfgp2\" (UniqueName: \"kubernetes.io/projected/458b3a36-a8cc-45a3-811f-7372033fa4b5-kube-api-access-rfgp2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:02:01.256589 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:01.256557    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="458b3a36-a8cc-45a3-811f-7372033fa4b5" path="/var/lib/kubelet/pods/458b3a36-a8cc-45a3-811f-7372033fa4b5/volumes"
Sep 12 03:02:01.256949 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:01.256933    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab45b64-ca8f-4b37-8dfe-7eb812836c1b" path="/var/lib/kubelet/pods/dab45b64-ca8f-4b37-8dfe-7eb812836c1b/volumes"
Sep 12 03:02:05.475340 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:05.475307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:02:17.813341 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813306    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"]
Sep 12 03:02:17.813755 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813741    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00631628-4ba9-4020-953a-0ac13656178d" containerName="maas-api"
Sep 12 03:02:17.813816 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813757    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00631628-4ba9-4020-953a-0ac13656178d" containerName="maas-api"
Sep 12 03:02:17.813816 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813767    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458b3a36-a8cc-45a3-811f-7372033fa4b5" containerName="main"
Sep 12 03:02:17.813816 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813772    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="458b3a36-a8cc-45a3-811f-7372033fa4b5" containerName="main"
Sep 12 03:02:17.813816 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813786    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab45b64-ca8f-4b37-8dfe-7eb812836c1b" containerName="main"
Sep 12 03:02:17.813816 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813791    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab45b64-ca8f-4b37-8dfe-7eb812836c1b" containerName="main"
Sep 12 03:02:17.813986 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="458b3a36-a8cc-45a3-811f-7372033fa4b5" containerName="main"
Sep 12 03:02:17.813986 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813853    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00631628-4ba9-4020-953a-0ac13656178d" containerName="maas-api"
Sep 12 03:02:17.813986 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.813860    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab45b64-ca8f-4b37-8dfe-7eb812836c1b" containerName="main"
Sep 12 03:02:17.816938 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.816921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.820067 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.820042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"kube-root-ca.crt\""
Sep 12 03:02:17.820067 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.820055    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"openshift-service-ca.crt\""
Sep 12 03:02:17.820273 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.820042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"default-dockercfg-f52m4\""
Sep 12 03:02:17.820273 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.820083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"auth-test-model-e401bd-kserve-self-signed-certs\""
Sep 12 03:02:17.826859 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.826834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"]
Sep 12 03:02:17.843149 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.843126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-dshm\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.843257 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.843162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-model-cache\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.843257 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.843183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-home\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.843333 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.843304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-tmp-dir\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.843368 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.843329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0488848d-2317-4f70-9237-5ae16daf1fbb-tls-certs\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.843403 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.843368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6bv\" (UniqueName: \"kubernetes.io/projected/0488848d-2317-4f70-9237-5ae16daf1fbb-kube-api-access-qf6bv\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944151 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-tmp-dir\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944322 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0488848d-2317-4f70-9237-5ae16daf1fbb-tls-certs\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944322 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6bv\" (UniqueName: \"kubernetes.io/projected/0488848d-2317-4f70-9237-5ae16daf1fbb-kube-api-access-qf6bv\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944322 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-dshm\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944322 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-model-cache\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944322 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-home\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944574 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-tmp-dir\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944690 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-model-cache\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.944806 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.944708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-home\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.946637 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.946615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-dshm\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.946893 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.946877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0488848d-2317-4f70-9237-5ae16daf1fbb-tls-certs\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:17.951356 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:17.951334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6bv\" (UniqueName: \"kubernetes.io/projected/0488848d-2317-4f70-9237-5ae16daf1fbb-kube-api-access-qf6bv\") pod \"auth-test-model-e401bd-kserve-7db746f5-l5gk6\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:18.126950 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:18.126878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:18.398481 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:18.398458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"]
Sep 12 03:02:18.400193 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:02:18.400161    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0488848d_2317_4f70_9237_5ae16daf1fbb.slice/crio-b4fd71389b490f518073b87967207ce29eff943109e4060ed3f254a8292daeee WatchSource:0}: Error finding container b4fd71389b490f518073b87967207ce29eff943109e4060ed3f254a8292daeee: Status 404 returned error can't find the container with id b4fd71389b490f518073b87967207ce29eff943109e4060ed3f254a8292daeee
Sep 12 03:02:18.828315 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:18.828274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6" event={"ID":"0488848d-2317-4f70-9237-5ae16daf1fbb","Type":"ContainerStarted","Data":"775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d"}
Sep 12 03:02:18.828729 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:18.828325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6" event={"ID":"0488848d-2317-4f70-9237-5ae16daf1fbb","Type":"ContainerStarted","Data":"b4fd71389b490f518073b87967207ce29eff943109e4060ed3f254a8292daeee"}
Sep 12 03:02:18.844486 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:18.844440    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6" podStartSLOduration=1.844427317 podStartE2EDuration="1.844427317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:02:18.844081904 +0000 UTC m=+1578.112115551" watchObservedRunningTime="2026-09-12 03:02:18.844427317 +0000 UTC m=+1578.112460964"
Sep 12 03:02:28.128002 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:28.127963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:28.128482 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:28.128021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:28.140344 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:28.140319    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:28.892821 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:28.892787    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:02:35.870065 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:35.870032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:02:42.716783 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.716748    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"]
Sep 12 03:02:42.720401 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.720383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.722807 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.722778    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"embedding-w0-b0573c-kserve-self-signed-certs\""
Sep 12 03:02:42.729208 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.728867    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"]
Sep 12 03:02:42.890003 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.889967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-home\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.890195 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.890015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcl6\" (UniqueName: \"kubernetes.io/projected/f5a213f8-3d1e-4670-8cec-56b5c0853758-kube-api-access-4gcl6\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.890195 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.890152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-dshm\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.890195 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.890182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a213f8-3d1e-4670-8cec-56b5c0853758-tls-certs\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.890371 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.890219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-model-cache\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.890371 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.890293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-tmp-dir\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.991861 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.991776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-home\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.991861 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.991822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcl6\" (UniqueName: \"kubernetes.io/projected/f5a213f8-3d1e-4670-8cec-56b5c0853758-kube-api-access-4gcl6\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.992118 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.991878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-dshm\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.992118 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.991898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a213f8-3d1e-4670-8cec-56b5c0853758-tls-certs\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.992118 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.991936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-model-cache\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.992118 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.991966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-tmp-dir\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.992288 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.992254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-home\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.992386 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.992360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-model-cache\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.992448 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.992409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-tmp-dir\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.994378 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.994361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-dshm\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.994765 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.994741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a213f8-3d1e-4670-8cec-56b5c0853758-tls-certs\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:42.999539 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:42.999516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcl6\" (UniqueName: \"kubernetes.io/projected/f5a213f8-3d1e-4670-8cec-56b5c0853758-kube-api-access-4gcl6\") pod \"embedding-w0-b0573c-kserve-6c4f474984-jdk6b\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") " pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:43.031527 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:43.031505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:43.301761 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:43.301597    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"]
Sep 12 03:02:43.304737 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:02:43.304702    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a213f8_3d1e_4670_8cec_56b5c0853758.slice/crio-8799d69a218e713ec8874412dff006f660aafd5720cb5565f9c6af4b00ef2749 WatchSource:0}: Error finding container 8799d69a218e713ec8874412dff006f660aafd5720cb5565f9c6af4b00ef2749: Status 404 returned error can't find the container with id 8799d69a218e713ec8874412dff006f660aafd5720cb5565f9c6af4b00ef2749
Sep 12 03:02:43.962624 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:43.962595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b" event={"ID":"f5a213f8-3d1e-4670-8cec-56b5c0853758","Type":"ContainerStarted","Data":"84d9a278effa3488b7f7d5ce3d96842a90067bb09972f47171f8b05f0d9b4ba5"}
Sep 12 03:02:43.963014 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:43.962633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b" event={"ID":"f5a213f8-3d1e-4670-8cec-56b5c0853758","Type":"ContainerStarted","Data":"8799d69a218e713ec8874412dff006f660aafd5720cb5565f9c6af4b00ef2749"}
Sep 12 03:02:43.979300 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:43.979247    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b" podStartSLOduration=1.979232103 podStartE2EDuration="1.979232103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:02:43.97694472 +0000 UTC m=+1603.244978390" watchObservedRunningTime="2026-09-12 03:02:43.979232103 +0000 UTC m=+1603.247265750"
Sep 12 03:02:53.032418 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:53.032329    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:53.032418 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:53.032408    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:53.044943 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:53.044908    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:02:54.029830 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:02:54.029795    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:03:12.661642 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:12.661610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:03:34.921586 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.921542    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"]
Sep 12 03:03:34.925329 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.925306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:34.927403 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.927381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-69d2db-serving-cert\""
Sep 12 03:03:34.933768 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.933748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"]
Sep 12 03:03:34.998356 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.998325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kr8\" (UniqueName: \"kubernetes.io/projected/28f99cad-7fd6-490f-b578-00d9742dde49-kube-api-access-c9kr8\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:34.998540 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.998370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-metrics-certs\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:34.998540 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.998470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-maas-api-tls\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:34.998540 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:34.998495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f99cad-7fd6-490f-b578-00d9742dde49-openshift-service-ca\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.099889 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.099846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-maas-api-tls\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.099889 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.099882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f99cad-7fd6-490f-b578-00d9742dde49-openshift-service-ca\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.100211 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.099942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kr8\" (UniqueName: \"kubernetes.io/projected/28f99cad-7fd6-490f-b578-00d9742dde49-kube-api-access-c9kr8\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.100211 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.099970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-metrics-certs\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.100805 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.100776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f99cad-7fd6-490f-b578-00d9742dde49-openshift-service-ca\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.102687 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.102656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-maas-api-tls\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.102841 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.102820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-metrics-certs\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.107197 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.107175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kr8\" (UniqueName: \"kubernetes.io/projected/28f99cad-7fd6-490f-b578-00d9742dde49-kube-api-access-c9kr8\") pod \"maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.239380 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.239349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:35.513383 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.513362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"]
Sep 12 03:03:35.515166 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:03:35.515141    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f99cad_7fd6_490f_b578_00d9742dde49.slice/crio-a1918f38fb462889664cdfb98befd7388fdaaf15cca66d0eb58043c91f31def4 WatchSource:0}: Error finding container a1918f38fb462889664cdfb98befd7388fdaaf15cca66d0eb58043c91f31def4: Status 404 returned error can't find the container with id a1918f38fb462889664cdfb98befd7388fdaaf15cca66d0eb58043c91f31def4
Sep 12 03:03:35.516903 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:35.516883    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 03:03:36.256044 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:36.256007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn" event={"ID":"28f99cad-7fd6-490f-b578-00d9742dde49","Type":"ContainerStarted","Data":"d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee"}
Sep 12 03:03:36.256447 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:36.256073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn" event={"ID":"28f99cad-7fd6-490f-b578-00d9742dde49","Type":"ContainerStarted","Data":"a1918f38fb462889664cdfb98befd7388fdaaf15cca66d0eb58043c91f31def4"}
Sep 12 03:03:36.256447 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:36.256204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:36.272825 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:36.272780    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn" podStartSLOduration=1.89020239 podStartE2EDuration="2.272765266s" totalImagesPullingTime="382.562876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:03:36.270346067 +0000 UTC m=+1655.538379717" watchObservedRunningTime="2026-09-12 03:03:36.272765266 +0000 UTC m=+1655.540798953"
Sep 12 03:03:42.266540 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:42.266510    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:03:48.151010 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:03:48.150978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:04:01.204530 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:01.204497    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"]
Sep 12 03:04:01.215448 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:01.215423    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"]
Sep 12 03:04:01.235479 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:01.235450    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"]
Sep 12 03:04:10.727242 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.727206    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"]
Sep 12 03:04:10.731994 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.731969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.734793 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.734762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-abddb9-serving-cert\""
Sep 12 03:04:10.749828 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.749801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"]
Sep 12 03:04:10.856688 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.856659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-metrics-certs\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.856853 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.856727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-openshift-service-ca\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.856853 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.856787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7n4\" (UniqueName: \"kubernetes.io/projected/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-kube-api-access-ms7n4\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.856853 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.856848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-maas-api-tls\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.958410 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.958376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-metrics-certs\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.958585 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.958472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-openshift-service-ca\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.958585 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.958505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7n4\" (UniqueName: \"kubernetes.io/projected/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-kube-api-access-ms7n4\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.958585 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.958538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-maas-api-tls\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.959177 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.959156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-openshift-service-ca\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.960937 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.960913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-maas-api-tls\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.961046 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.961009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-metrics-certs\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:10.966941 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:10.966923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7n4\" (UniqueName: \"kubernetes.io/projected/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-kube-api-access-ms7n4\") pod \"maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:11.046491 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:11.046421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:11.330622 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:11.330594    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"]
Sep 12 03:04:11.332847 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:04:11.332820    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65a39b4_c76d_49ef_8a23_0a7649d1ae41.slice/crio-0bbe3299589b30e2f17e83ef9dc2b34d936c3169b49a834e50965bbf6cd1b54a WatchSource:0}: Error finding container 0bbe3299589b30e2f17e83ef9dc2b34d936c3169b49a834e50965bbf6cd1b54a: Status 404 returned error can't find the container with id 0bbe3299589b30e2f17e83ef9dc2b34d936c3169b49a834e50965bbf6cd1b54a
Sep 12 03:04:11.442454 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:11.442411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9" event={"ID":"f65a39b4-c76d-49ef-8a23-0a7649d1ae41","Type":"ContainerStarted","Data":"0bbe3299589b30e2f17e83ef9dc2b34d936c3169b49a834e50965bbf6cd1b54a"}
Sep 12 03:04:12.449670 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:12.449638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9" event={"ID":"f65a39b4-c76d-49ef-8a23-0a7649d1ae41","Type":"ContainerStarted","Data":"2d8181b6d96fbfe30333124ed723f7b75f8d76df74435e10cc840cff19264bc8"}
Sep 12 03:04:12.450262 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:12.449760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:12.470291 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:12.470247    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9" podStartSLOduration=1.929952968 podStartE2EDuration="2.470233977s" totalImagesPullingTime="540.281009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:04:12.468658766 +0000 UTC m=+1691.736692412" watchObservedRunningTime="2026-09-12 03:04:12.470233977 +0000 UTC m=+1691.738267624"
Sep 12 03:04:16.225625 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.225584    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b" podUID="f5a213f8-3d1e-4670-8cec-56b5c0853758" containerName="main" containerID="cri-o://84d9a278effa3488b7f7d5ce3d96842a90067bb09972f47171f8b05f0d9b4ba5" gracePeriod=45
Sep 12 03:04:16.236528 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.236496    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm" podUID="b7a71477-fd5b-47ed-9949-a30143d02142" containerName="main" containerID="cri-o://aca946bbd79525356307e625adf408addaec93801c5145bc0ed532b0588ac25e" gracePeriod=45
Sep 12 03:04:16.254793 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.254760    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf" podUID="e89c3a97-7694-47be-818f-eeb429d25743" containerName="main" containerID="cri-o://f68a866c8d54bd359b028779c5f132bbfcb5b0bba9c749de425f1af10371e2ab" gracePeriod=45
Sep 12 03:04:16.477843 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.477743    2579 generic.go:356] "Generic (PLEG): container finished" podID="e89c3a97-7694-47be-818f-eeb429d25743" containerID="f68a866c8d54bd359b028779c5f132bbfcb5b0bba9c749de425f1af10371e2ab" exitCode=0
Sep 12 03:04:16.477843 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.477823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf" event={"ID":"e89c3a97-7694-47be-818f-eeb429d25743","Type":"ContainerDied","Data":"f68a866c8d54bd359b028779c5f132bbfcb5b0bba9c749de425f1af10371e2ab"}
Sep 12 03:04:16.480479 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.480449    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5a213f8-3d1e-4670-8cec-56b5c0853758" containerID="84d9a278effa3488b7f7d5ce3d96842a90067bb09972f47171f8b05f0d9b4ba5" exitCode=0
Sep 12 03:04:16.480591 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.480542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b" event={"ID":"f5a213f8-3d1e-4670-8cec-56b5c0853758","Type":"ContainerDied","Data":"84d9a278effa3488b7f7d5ce3d96842a90067bb09972f47171f8b05f0d9b4ba5"}
Sep 12 03:04:16.482394 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.482372    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7a71477-fd5b-47ed-9949-a30143d02142" containerID="aca946bbd79525356307e625adf408addaec93801c5145bc0ed532b0588ac25e" exitCode=0
Sep 12 03:04:16.482489 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.482448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm" event={"ID":"b7a71477-fd5b-47ed-9949-a30143d02142","Type":"ContainerDied","Data":"aca946bbd79525356307e625adf408addaec93801c5145bc0ed532b0588ac25e"}
Sep 12 03:04:16.564489 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.564468    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:04:16.568246 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.568160    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 03:04:16.573152 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.573132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 03:04:16.613118 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.612694    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"]
Sep 12 03:04:16.616850 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.616801    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b0573c-29819700-nvkdk"]
Sep 12 03:04:16.618693 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.618671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-tmp-dir\") pod \"b7a71477-fd5b-47ed-9949-a30143d02142\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") "
Sep 12 03:04:16.619151 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7a71477-fd5b-47ed-9949-a30143d02142" (UID: "b7a71477-fd5b-47ed-9949-a30143d02142"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.619387 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gcl6\" (UniqueName: \"kubernetes.io/projected/f5a213f8-3d1e-4670-8cec-56b5c0853758-kube-api-access-4gcl6\") pod \"f5a213f8-3d1e-4670-8cec-56b5c0853758\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") "
Sep 12 03:04:16.619512 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a71477-fd5b-47ed-9949-a30143d02142-tls-certs\") pod \"b7a71477-fd5b-47ed-9949-a30143d02142\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") "
Sep 12 03:04:16.619512 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619496    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-home\") pod \"f5a213f8-3d1e-4670-8cec-56b5c0853758\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") "
Sep 12 03:04:16.619648 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-model-cache\") pod \"f5a213f8-3d1e-4670-8cec-56b5c0853758\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") "
Sep 12 03:04:16.619648 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-tmp-dir\") pod \"f5a213f8-3d1e-4670-8cec-56b5c0853758\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") "
Sep 12 03:04:16.619648 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a213f8-3d1e-4670-8cec-56b5c0853758-tls-certs\") pod \"f5a213f8-3d1e-4670-8cec-56b5c0853758\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") "
Sep 12 03:04:16.619821 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-home\") pod \"b7a71477-fd5b-47ed-9949-a30143d02142\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") "
Sep 12 03:04:16.619821 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619684    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-model-cache\") pod \"b7a71477-fd5b-47ed-9949-a30143d02142\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") "
Sep 12 03:04:16.619821 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-dshm\") pod \"f5a213f8-3d1e-4670-8cec-56b5c0853758\" (UID: \"f5a213f8-3d1e-4670-8cec-56b5c0853758\") "
Sep 12 03:04:16.619821 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdbm6\" (UniqueName: \"kubernetes.io/projected/b7a71477-fd5b-47ed-9949-a30143d02142-kube-api-access-tdbm6\") pod \"b7a71477-fd5b-47ed-9949-a30143d02142\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") "
Sep 12 03:04:16.619821 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.619800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-dshm\") pod \"b7a71477-fd5b-47ed-9949-a30143d02142\" (UID: \"b7a71477-fd5b-47ed-9949-a30143d02142\") "
Sep 12 03:04:16.620157 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.620129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-home" (OuterVolumeSpecName: "home") pod "f5a213f8-3d1e-4670-8cec-56b5c0853758" (UID: "f5a213f8-3d1e-4670-8cec-56b5c0853758"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.620352 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.620322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7a71477-fd5b-47ed-9949-a30143d02142" (UID: "b7a71477-fd5b-47ed-9949-a30143d02142"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.620720 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.620692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5a213f8-3d1e-4670-8cec-56b5c0853758" (UID: "f5a213f8-3d1e-4670-8cec-56b5c0853758"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.620822 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.620722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5a213f8-3d1e-4670-8cec-56b5c0853758" (UID: "f5a213f8-3d1e-4670-8cec-56b5c0853758"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.620941 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.620921    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-home" (OuterVolumeSpecName: "home") pod "b7a71477-fd5b-47ed-9949-a30143d02142" (UID: "b7a71477-fd5b-47ed-9949-a30143d02142"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.621474 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.621449    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.621571 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.621478    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.621571 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.621494    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.621571 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.621506    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.621571 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.621518    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.621571 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.621531    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.623832 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.623798    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a213f8-3d1e-4670-8cec-56b5c0853758-kube-api-access-4gcl6" (OuterVolumeSpecName: "kube-api-access-4gcl6") pod "f5a213f8-3d1e-4670-8cec-56b5c0853758" (UID: "f5a213f8-3d1e-4670-8cec-56b5c0853758"). InnerVolumeSpecName "kube-api-access-4gcl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:16.624994 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.624966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-dshm" (OuterVolumeSpecName: "dshm") pod "b7a71477-fd5b-47ed-9949-a30143d02142" (UID: "b7a71477-fd5b-47ed-9949-a30143d02142"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.625991 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.625786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a213f8-3d1e-4670-8cec-56b5c0853758-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5a213f8-3d1e-4670-8cec-56b5c0853758" (UID: "f5a213f8-3d1e-4670-8cec-56b5c0853758"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:16.625991 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.625863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-dshm" (OuterVolumeSpecName: "dshm") pod "f5a213f8-3d1e-4670-8cec-56b5c0853758" (UID: "f5a213f8-3d1e-4670-8cec-56b5c0853758"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.626775 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.626747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a71477-fd5b-47ed-9949-a30143d02142-kube-api-access-tdbm6" (OuterVolumeSpecName: "kube-api-access-tdbm6") pod "b7a71477-fd5b-47ed-9949-a30143d02142" (UID: "b7a71477-fd5b-47ed-9949-a30143d02142"). InnerVolumeSpecName "kube-api-access-tdbm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:16.627722 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.627702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a71477-fd5b-47ed-9949-a30143d02142-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7a71477-fd5b-47ed-9949-a30143d02142" (UID: "b7a71477-fd5b-47ed-9949-a30143d02142"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:16.722263 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-tmp-dir\") pod \"e89c3a97-7694-47be-818f-eeb429d25743\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") "
Sep 12 03:04:16.722263 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722268    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-dshm\") pod \"e89c3a97-7694-47be-818f-eeb429d25743\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") "
Sep 12 03:04:16.722482 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722302    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-home\") pod \"e89c3a97-7694-47be-818f-eeb429d25743\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") "
Sep 12 03:04:16.722482 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89c3a97-7694-47be-818f-eeb429d25743-tls-certs\") pod \"e89c3a97-7694-47be-818f-eeb429d25743\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") "
Sep 12 03:04:16.722482 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-model-cache\") pod \"e89c3a97-7694-47be-818f-eeb429d25743\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") "
Sep 12 03:04:16.722482 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnq6n\" (UniqueName: \"kubernetes.io/projected/e89c3a97-7694-47be-818f-eeb429d25743-kube-api-access-wnq6n\") pod \"e89c3a97-7694-47be-818f-eeb429d25743\" (UID: \"e89c3a97-7694-47be-818f-eeb429d25743\") "
Sep 12 03:04:16.722672 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e89c3a97-7694-47be-818f-eeb429d25743" (UID: "e89c3a97-7694-47be-818f-eeb429d25743"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.722672 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722617    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-home" (OuterVolumeSpecName: "home") pod "e89c3a97-7694-47be-818f-eeb429d25743" (UID: "e89c3a97-7694-47be-818f-eeb429d25743"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.722780 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-model-cache" (OuterVolumeSpecName: "model-cache") pod "e89c3a97-7694-47be-818f-eeb429d25743" (UID: "e89c3a97-7694-47be-818f-eeb429d25743"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.722974 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722950    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a213f8-3d1e-4670-8cec-56b5c0853758-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.722974 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722975    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.723357 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.722988    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.723357 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.723000    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a213f8-3d1e-4670-8cec-56b5c0853758-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.723357 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.723013    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbm6\" (UniqueName: \"kubernetes.io/projected/b7a71477-fd5b-47ed-9949-a30143d02142-kube-api-access-tdbm6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.723357 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.723026    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a71477-fd5b-47ed-9949-a30143d02142-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.723357 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.723039    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcl6\" (UniqueName: \"kubernetes.io/projected/f5a213f8-3d1e-4670-8cec-56b5c0853758-kube-api-access-4gcl6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.723357 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.723051    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.723357 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.723064    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a71477-fd5b-47ed-9949-a30143d02142-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.724555 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.724530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-dshm" (OuterVolumeSpecName: "dshm") pod "e89c3a97-7694-47be-818f-eeb429d25743" (UID: "e89c3a97-7694-47be-818f-eeb429d25743"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.724749 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.724728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89c3a97-7694-47be-818f-eeb429d25743-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e89c3a97-7694-47be-818f-eeb429d25743" (UID: "e89c3a97-7694-47be-818f-eeb429d25743"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:16.724836 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.724818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89c3a97-7694-47be-818f-eeb429d25743-kube-api-access-wnq6n" (OuterVolumeSpecName: "kube-api-access-wnq6n") pod "e89c3a97-7694-47be-818f-eeb429d25743" (UID: "e89c3a97-7694-47be-818f-eeb429d25743"). InnerVolumeSpecName "kube-api-access-wnq6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:16.824576 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.824510    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89c3a97-7694-47be-818f-eeb429d25743-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.824576 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.824534    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89c3a97-7694-47be-818f-eeb429d25743-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.824576 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:16.824546    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnq6n\" (UniqueName: \"kubernetes.io/projected/e89c3a97-7694-47be-818f-eeb429d25743-kube-api-access-wnq6n\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:17.256513 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.256478    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b881678-7bd6-49bf-b67d-92f4fe9dab9c" path="/var/lib/kubelet/pods/8b881678-7bd6-49bf-b67d-92f4fe9dab9c/volumes"
Sep 12 03:04:17.490119 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.490003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b" event={"ID":"f5a213f8-3d1e-4670-8cec-56b5c0853758","Type":"ContainerDied","Data":"8799d69a218e713ec8874412dff006f660aafd5720cb5565f9c6af4b00ef2749"}
Sep 12 03:04:17.490119 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.490029    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"
Sep 12 03:04:17.490119 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.490066    2579 scope.go:117] "RemoveContainer" containerID="84d9a278effa3488b7f7d5ce3d96842a90067bb09972f47171f8b05f0d9b4ba5"
Sep 12 03:04:17.491809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.491614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm" event={"ID":"b7a71477-fd5b-47ed-9949-a30143d02142","Type":"ContainerDied","Data":"93640c53817c970e1c233f020db33388e794728fad01fcf8e65831736f0f14dc"}
Sep 12 03:04:17.491809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.491695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"
Sep 12 03:04:17.493906 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.493879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf" event={"ID":"e89c3a97-7694-47be-818f-eeb429d25743","Type":"ContainerDied","Data":"d049226e089d98a7328f823c00fb01616c7e628e79ddda713fd2cb66e9263b02"}
Sep 12 03:04:17.494087 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.493896    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"
Sep 12 03:04:17.501259 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.501244    2579 scope.go:117] "RemoveContainer" containerID="aca946bbd79525356307e625adf408addaec93801c5145bc0ed532b0588ac25e"
Sep 12 03:04:17.514545 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.514526    2579 scope.go:117] "RemoveContainer" containerID="f68a866c8d54bd359b028779c5f132bbfcb5b0bba9c749de425f1af10371e2ab"
Sep 12 03:04:17.523712 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.523693    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"]
Sep 12 03:04:17.529544 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.529522    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/simulator-w0-b0573c-kserve-5b9d779465-j8rkf"]
Sep 12 03:04:17.543752 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.543729    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"]
Sep 12 03:04:17.547235 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.547215    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/embedding-w0-b0573c-kserve-6c4f474984-jdk6b"]
Sep 12 03:04:17.566057 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.566036    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"]
Sep 12 03:04:17.569609 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:17.569588    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/premium-w0-b0573c-kserve-55c76788d4-89gmm"]
Sep 12 03:04:18.461375 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:18.461348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:04:19.250103 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:19.250070    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a71477-fd5b-47ed-9949-a30143d02142" path="/var/lib/kubelet/pods/b7a71477-fd5b-47ed-9949-a30143d02142/volumes"
Sep 12 03:04:19.250492 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:19.250475    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89c3a97-7694-47be-818f-eeb429d25743" path="/var/lib/kubelet/pods/e89c3a97-7694-47be-818f-eeb429d25743/volumes"
Sep 12 03:04:19.250828 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:19.250816    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a213f8-3d1e-4670-8cec-56b5c0853758" path="/var/lib/kubelet/pods/f5a213f8-3d1e-4670-8cec-56b5c0853758/volumes"
Sep 12 03:04:39.847402 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:39.847323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:04:48.823062 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:48.823027    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"]
Sep 12 03:04:48.823719 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:48.823306    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn" podUID="28f99cad-7fd6-490f-b578-00d9742dde49" containerName="maas-api" containerID="cri-o://d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee" gracePeriod=30
Sep 12 03:04:49.067033 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.067009    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:04:49.132590 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.132513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-metrics-certs\") pod \"28f99cad-7fd6-490f-b578-00d9742dde49\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") "
Sep 12 03:04:49.132590 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.132574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f99cad-7fd6-490f-b578-00d9742dde49-openshift-service-ca\") pod \"28f99cad-7fd6-490f-b578-00d9742dde49\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") "
Sep 12 03:04:49.132797 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.132703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9kr8\" (UniqueName: \"kubernetes.io/projected/28f99cad-7fd6-490f-b578-00d9742dde49-kube-api-access-c9kr8\") pod \"28f99cad-7fd6-490f-b578-00d9742dde49\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") "
Sep 12 03:04:49.132797 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.132738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-maas-api-tls\") pod \"28f99cad-7fd6-490f-b578-00d9742dde49\" (UID: \"28f99cad-7fd6-490f-b578-00d9742dde49\") "
Sep 12 03:04:49.133011 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.132982    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f99cad-7fd6-490f-b578-00d9742dde49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28f99cad-7fd6-490f-b578-00d9742dde49" (UID: "28f99cad-7fd6-490f-b578-00d9742dde49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:04:49.134765 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.134741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28f99cad-7fd6-490f-b578-00d9742dde49" (UID: "28f99cad-7fd6-490f-b578-00d9742dde49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:49.134923 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.134902    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28f99cad-7fd6-490f-b578-00d9742dde49" (UID: "28f99cad-7fd6-490f-b578-00d9742dde49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:49.135006 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.134992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f99cad-7fd6-490f-b578-00d9742dde49-kube-api-access-c9kr8" (OuterVolumeSpecName: "kube-api-access-c9kr8") pod "28f99cad-7fd6-490f-b578-00d9742dde49" (UID: "28f99cad-7fd6-490f-b578-00d9742dde49"). InnerVolumeSpecName "kube-api-access-c9kr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:49.233846 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.233818    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9kr8\" (UniqueName: \"kubernetes.io/projected/28f99cad-7fd6-490f-b578-00d9742dde49-kube-api-access-c9kr8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:49.233846 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.233842    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:49.233846 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.233850    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f99cad-7fd6-490f-b578-00d9742dde49-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:49.234053 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.233860    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f99cad-7fd6-490f-b578-00d9742dde49-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:04:49.671085 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.671044    2579 generic.go:356] "Generic (PLEG): container finished" podID="28f99cad-7fd6-490f-b578-00d9742dde49" containerID="d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee" exitCode=0
Sep 12 03:04:49.671288 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.671106    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"
Sep 12 03:04:49.671288 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.671137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn" event={"ID":"28f99cad-7fd6-490f-b578-00d9742dde49","Type":"ContainerDied","Data":"d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee"}
Sep 12 03:04:49.671288 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.671192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn" event={"ID":"28f99cad-7fd6-490f-b578-00d9742dde49","Type":"ContainerDied","Data":"a1918f38fb462889664cdfb98befd7388fdaaf15cca66d0eb58043c91f31def4"}
Sep 12 03:04:49.671288 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.671219    2579 scope.go:117] "RemoveContainer" containerID="d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee"
Sep 12 03:04:49.682819 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.682796    2579 scope.go:117] "RemoveContainer" containerID="d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee"
Sep 12 03:04:49.683115 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:04:49.683080    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee\": container with ID starting with d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee not found: ID does not exist" containerID="d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee"
Sep 12 03:04:49.683189 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.683124    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee"} err="failed to get container status \"d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee\": rpc error: code = NotFound desc = could not find container \"d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee\": container with ID starting with d8a1a296af501eac00bcbdcda019a674f73bc22e6354fafe93ffff20c69b78ee not found: ID does not exist"
Sep 12 03:04:49.687838 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.687815    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"]
Sep 12 03:04:49.691062 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:49.691040    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69d2db-6d87cbfd4c-vrqtn"]
Sep 12 03:04:51.251480 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:04:51.251439    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28f99cad-7fd6-490f-b578-00d9742dde49" path="/var/lib/kubelet/pods/28f99cad-7fd6-490f-b578-00d9742dde49/volumes"
Sep 12 03:05:08.045000 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:08.044963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:05:11.549157 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.549125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"]
Sep 12 03:05:11.549589 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.549382    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9" podUID="f65a39b4-c76d-49ef-8a23-0a7649d1ae41" containerName="maas-api" containerID="cri-o://2d8181b6d96fbfe30333124ed723f7b75f8d76df74435e10cc840cff19264bc8" gracePeriod=30
Sep 12 03:05:11.790574 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.790528    2579 generic.go:356] "Generic (PLEG): container finished" podID="f65a39b4-c76d-49ef-8a23-0a7649d1ae41" containerID="2d8181b6d96fbfe30333124ed723f7b75f8d76df74435e10cc840cff19264bc8" exitCode=0
Sep 12 03:05:11.790711 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.790607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9" event={"ID":"f65a39b4-c76d-49ef-8a23-0a7649d1ae41","Type":"ContainerDied","Data":"2d8181b6d96fbfe30333124ed723f7b75f8d76df74435e10cc840cff19264bc8"}
Sep 12 03:05:11.812464 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.812403    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:05:11.844954 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.844922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7n4\" (UniqueName: \"kubernetes.io/projected/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-kube-api-access-ms7n4\") pod \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") "
Sep 12 03:05:11.845143 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.844988    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-metrics-certs\") pod \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") "
Sep 12 03:05:11.845143 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.845010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-maas-api-tls\") pod \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") "
Sep 12 03:05:11.845143 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.845062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-openshift-service-ca\") pod \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\" (UID: \"f65a39b4-c76d-49ef-8a23-0a7649d1ae41\") "
Sep 12 03:05:11.845401 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.845373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f65a39b4-c76d-49ef-8a23-0a7649d1ae41" (UID: "f65a39b4-c76d-49ef-8a23-0a7649d1ae41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:05:11.845532 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.845486    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:11.847463 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.847441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f65a39b4-c76d-49ef-8a23-0a7649d1ae41" (UID: "f65a39b4-c76d-49ef-8a23-0a7649d1ae41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:05:11.847734 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.847711    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f65a39b4-c76d-49ef-8a23-0a7649d1ae41" (UID: "f65a39b4-c76d-49ef-8a23-0a7649d1ae41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:05:11.847790 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.847731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-kube-api-access-ms7n4" (OuterVolumeSpecName: "kube-api-access-ms7n4") pod "f65a39b4-c76d-49ef-8a23-0a7649d1ae41" (UID: "f65a39b4-c76d-49ef-8a23-0a7649d1ae41"). InnerVolumeSpecName "kube-api-access-ms7n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:05:11.946064 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.946016    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:11.946064 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.946057    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:11.946064 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:11.946072    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7n4\" (UniqueName: \"kubernetes.io/projected/f65a39b4-c76d-49ef-8a23-0a7649d1ae41-kube-api-access-ms7n4\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:12.798643 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:12.798609    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"
Sep 12 03:05:12.799078 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:12.798655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9" event={"ID":"f65a39b4-c76d-49ef-8a23-0a7649d1ae41","Type":"ContainerDied","Data":"0bbe3299589b30e2f17e83ef9dc2b34d936c3169b49a834e50965bbf6cd1b54a"}
Sep 12 03:05:12.799078 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:12.798730    2579 scope.go:117] "RemoveContainer" containerID="2d8181b6d96fbfe30333124ed723f7b75f8d76df74435e10cc840cff19264bc8"
Sep 12 03:05:12.821023 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:12.820923    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"]
Sep 12 03:05:12.822894 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:12.822864    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abddb9-5d5c44df95-c8hk9"]
Sep 12 03:05:13.249168 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:13.249140    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65a39b4-c76d-49ef-8a23-0a7649d1ae41" path="/var/lib/kubelet/pods/f65a39b4-c76d-49ef-8a23-0a7649d1ae41/volumes"
Sep 12 03:05:29.108982 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.108952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"]
Sep 12 03:05:29.109461 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.109228    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6" podUID="58bac08d-5a46-48c4-b1e0-3b433ebf27ed" containerName="maas-api" containerID="cri-o://80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28" gracePeriod=30
Sep 12 03:05:29.361288 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.361237    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 03:05:29.407225 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.407201    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-openshift-service-ca\") pod \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") "
Sep 12 03:05:29.407394 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.407246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb7tm\" (UniqueName: \"kubernetes.io/projected/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-kube-api-access-gb7tm\") pod \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") "
Sep 12 03:05:29.407394 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.407270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-maas-api-tls\") pod \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") "
Sep 12 03:05:29.407394 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.407308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-metrics-certs\") pod \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\" (UID: \"58bac08d-5a46-48c4-b1e0-3b433ebf27ed\") "
Sep 12 03:05:29.407653 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.407625    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58bac08d-5a46-48c4-b1e0-3b433ebf27ed" (UID: "58bac08d-5a46-48c4-b1e0-3b433ebf27ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:05:29.409601 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.409573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "58bac08d-5a46-48c4-b1e0-3b433ebf27ed" (UID: "58bac08d-5a46-48c4-b1e0-3b433ebf27ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:05:29.409703 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.409615    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-kube-api-access-gb7tm" (OuterVolumeSpecName: "kube-api-access-gb7tm") pod "58bac08d-5a46-48c4-b1e0-3b433ebf27ed" (UID: "58bac08d-5a46-48c4-b1e0-3b433ebf27ed"). InnerVolumeSpecName "kube-api-access-gb7tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:05:29.409747 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.409717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58bac08d-5a46-48c4-b1e0-3b433ebf27ed" (UID: "58bac08d-5a46-48c4-b1e0-3b433ebf27ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:05:29.508920 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.508886    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:29.508920 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.508918    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7tm\" (UniqueName: \"kubernetes.io/projected/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-kube-api-access-gb7tm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:29.509128 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.508932    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:29.509128 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.508943    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58bac08d-5a46-48c4-b1e0-3b433ebf27ed-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:05:29.886000 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.885965    2579 generic.go:356] "Generic (PLEG): container finished" podID="58bac08d-5a46-48c4-b1e0-3b433ebf27ed" containerID="80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28" exitCode=0
Sep 12 03:05:29.886197 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.886017    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"
Sep 12 03:05:29.886197 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.886053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6" event={"ID":"58bac08d-5a46-48c4-b1e0-3b433ebf27ed","Type":"ContainerDied","Data":"80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28"}
Sep 12 03:05:29.886197 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.886135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6" event={"ID":"58bac08d-5a46-48c4-b1e0-3b433ebf27ed","Type":"ContainerDied","Data":"5505cae9624e47e5953d4f4c14208881fec4d53697fcc7a87c7f954228b44242"}
Sep 12 03:05:29.886197 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.886158    2579 scope.go:117] "RemoveContainer" containerID="80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28"
Sep 12 03:05:29.896665 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.896645    2579 scope.go:117] "RemoveContainer" containerID="80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28"
Sep 12 03:05:29.896962 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:05:29.896939    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28\": container with ID starting with 80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28 not found: ID does not exist" containerID="80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28"
Sep 12 03:05:29.897072 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.896972    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28"} err="failed to get container status \"80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28\": rpc error: code = NotFound desc = could not find container \"80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28\": container with ID starting with 80aa0eb3dbfde9b79abc63a5bd31915aef89c3d939752d32848c47e5f83a5d28 not found: ID does not exist"
Sep 12 03:05:29.906681 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.906656    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"]
Sep 12 03:05:29.914450 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:29.914430    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-933bba-7d9546957-8rtj6"]
Sep 12 03:05:31.249505 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:31.249472    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58bac08d-5a46-48c4-b1e0-3b433ebf27ed" path="/var/lib/kubelet/pods/58bac08d-5a46-48c4-b1e0-3b433ebf27ed/volumes"
Sep 12 03:05:32.141277 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:32.141244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:05:54.596863 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:54.596779    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"]
Sep 12 03:05:54.606383 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:54.606358    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"]
Sep 12 03:05:58.348006 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:05:58.347977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:06:05.590743 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.590710    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"]
Sep 12 03:06:05.591156 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.590919    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2" podUID="451557bb-4bca-4342-b716-e1c95b08545c" containerName="maas-api" containerID="cri-o://e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d" gracePeriod=30
Sep 12 03:06:05.838187 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.838162    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 03:06:05.960169 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.960127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-maas-api-tls\") pod \"451557bb-4bca-4342-b716-e1c95b08545c\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") "
Sep 12 03:06:05.960333 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.960181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gsc8\" (UniqueName: \"kubernetes.io/projected/451557bb-4bca-4342-b716-e1c95b08545c-kube-api-access-6gsc8\") pod \"451557bb-4bca-4342-b716-e1c95b08545c\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") "
Sep 12 03:06:05.960400 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.960367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-metrics-certs\") pod \"451557bb-4bca-4342-b716-e1c95b08545c\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") "
Sep 12 03:06:05.960482 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.960470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451557bb-4bca-4342-b716-e1c95b08545c-openshift-service-ca\") pod \"451557bb-4bca-4342-b716-e1c95b08545c\" (UID: \"451557bb-4bca-4342-b716-e1c95b08545c\") "
Sep 12 03:06:05.960827 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.960802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451557bb-4bca-4342-b716-e1c95b08545c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "451557bb-4bca-4342-b716-e1c95b08545c" (UID: "451557bb-4bca-4342-b716-e1c95b08545c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:06:05.962574 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.962549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "451557bb-4bca-4342-b716-e1c95b08545c" (UID: "451557bb-4bca-4342-b716-e1c95b08545c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:05.962684 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.962572    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "451557bb-4bca-4342-b716-e1c95b08545c" (UID: "451557bb-4bca-4342-b716-e1c95b08545c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:05.962753 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:05.962712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451557bb-4bca-4342-b716-e1c95b08545c-kube-api-access-6gsc8" (OuterVolumeSpecName: "kube-api-access-6gsc8") pod "451557bb-4bca-4342-b716-e1c95b08545c" (UID: "451557bb-4bca-4342-b716-e1c95b08545c"). InnerVolumeSpecName "kube-api-access-6gsc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:06.061509 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.061481    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451557bb-4bca-4342-b716-e1c95b08545c-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:06.061509 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.061508    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:06.061711 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.061517    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gsc8\" (UniqueName: \"kubernetes.io/projected/451557bb-4bca-4342-b716-e1c95b08545c-kube-api-access-6gsc8\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:06.061711 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.061526    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451557bb-4bca-4342-b716-e1c95b08545c-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:06.079906 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.079874    2579 generic.go:356] "Generic (PLEG): container finished" podID="451557bb-4bca-4342-b716-e1c95b08545c" containerID="e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d" exitCode=0
Sep 12 03:06:06.080058 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.079938    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"
Sep 12 03:06:06.080058 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.079959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2" event={"ID":"451557bb-4bca-4342-b716-e1c95b08545c","Type":"ContainerDied","Data":"e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d"}
Sep 12 03:06:06.080058 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.080040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2" event={"ID":"451557bb-4bca-4342-b716-e1c95b08545c","Type":"ContainerDied","Data":"88ec991070dc96d77772e3fe26f56595ce95fe77d7e4c007511459bf5b095eee"}
Sep 12 03:06:06.080192 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.080062    2579 scope.go:117] "RemoveContainer" containerID="e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d"
Sep 12 03:06:06.090306 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.090288    2579 scope.go:117] "RemoveContainer" containerID="e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d"
Sep 12 03:06:06.090677 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:06.090655    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d\": container with ID starting with e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d not found: ID does not exist" containerID="e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d"
Sep 12 03:06:06.090744 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.090687    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d"} err="failed to get container status \"e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d\": rpc error: code = NotFound desc = could not find container \"e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d\": container with ID starting with e1c01df1f34d7572a48ba8f7d0585217397541d68b46f20b61af6ef2251fa88d not found: ID does not exist"
Sep 12 03:06:06.101433 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.101407    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"]
Sep 12 03:06:06.104581 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:06.104561    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c6190b-57bb8689f9-qttq2"]
Sep 12 03:06:07.249830 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:07.249792    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451557bb-4bca-4342-b716-e1c95b08545c" path="/var/lib/kubelet/pods/451557bb-4bca-4342-b716-e1c95b08545c/volumes"
Sep 12 03:06:09.618258 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:09.618174    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2" podUID="6c5751d3-515c-428b-9c76-cbdc7c7e4a12" containerName="main" containerID="cri-o://b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc" gracePeriod=45
Sep 12 03:06:09.625889 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:09.625853    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7" podUID="0217ee9c-e6ad-4b45-b748-3fda3e9cc791" containerName="main" containerID="cri-o://b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700" gracePeriod=45
Sep 12 03:06:09.866255 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:09.866217    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 03:06:09.893913 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:09.893888    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 03:06:10.001469 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tmp-dir\") pod \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") "
Sep 12 03:06:10.001469 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001475    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-dshm\") pod \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tls-certs\") pod \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-home\") pod \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-home\") pod \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tmp-dir\") pod \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tls-certs\") pod \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-model-cache\") pod \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-model-cache\") pod \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") "
Sep 12 03:06:10.001725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m78r\" (UniqueName: \"kubernetes.io/projected/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-kube-api-access-6m78r\") pod \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\" (UID: \"6c5751d3-515c-428b-9c76-cbdc7c7e4a12\") "
Sep 12 03:06:10.002107 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mlsr\" (UniqueName: \"kubernetes.io/projected/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-kube-api-access-9mlsr\") pod \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") "
Sep 12 03:06:10.002107 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c5751d3-515c-428b-9c76-cbdc7c7e4a12" (UID: "6c5751d3-515c-428b-9c76-cbdc7c7e4a12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.002107 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.001787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-dshm\") pod \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\" (UID: \"0217ee9c-e6ad-4b45-b748-3fda3e9cc791\") "
Sep 12 03:06:10.002257 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.002239    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.002479 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.002456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-home" (OuterVolumeSpecName: "home") pod "6c5751d3-515c-428b-9c76-cbdc7c7e4a12" (UID: "6c5751d3-515c-428b-9c76-cbdc7c7e4a12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.002655 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.002634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-home" (OuterVolumeSpecName: "home") pod "0217ee9c-e6ad-4b45-b748-3fda3e9cc791" (UID: "0217ee9c-e6ad-4b45-b748-3fda3e9cc791"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.002812 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.002791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c5751d3-515c-428b-9c76-cbdc7c7e4a12" (UID: "6c5751d3-515c-428b-9c76-cbdc7c7e4a12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.002967 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.002945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-model-cache" (OuterVolumeSpecName: "model-cache") pod "0217ee9c-e6ad-4b45-b748-3fda3e9cc791" (UID: "0217ee9c-e6ad-4b45-b748-3fda3e9cc791"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.004228 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.004185    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0217ee9c-e6ad-4b45-b748-3fda3e9cc791" (UID: "0217ee9c-e6ad-4b45-b748-3fda3e9cc791"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.004551 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.004513    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-dshm" (OuterVolumeSpecName: "dshm") pod "0217ee9c-e6ad-4b45-b748-3fda3e9cc791" (UID: "0217ee9c-e6ad-4b45-b748-3fda3e9cc791"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.004551 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.004517    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0217ee9c-e6ad-4b45-b748-3fda3e9cc791" (UID: "0217ee9c-e6ad-4b45-b748-3fda3e9cc791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:10.005799 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.005773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-kube-api-access-6m78r" (OuterVolumeSpecName: "kube-api-access-6m78r") pod "6c5751d3-515c-428b-9c76-cbdc7c7e4a12" (UID: "6c5751d3-515c-428b-9c76-cbdc7c7e4a12"). InnerVolumeSpecName "kube-api-access-6m78r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:10.005799 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.005806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-kube-api-access-9mlsr" (OuterVolumeSpecName: "kube-api-access-9mlsr") pod "0217ee9c-e6ad-4b45-b748-3fda3e9cc791" (UID: "0217ee9c-e6ad-4b45-b748-3fda3e9cc791"). InnerVolumeSpecName "kube-api-access-9mlsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:10.005996 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.005984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c5751d3-515c-428b-9c76-cbdc7c7e4a12" (UID: "6c5751d3-515c-428b-9c76-cbdc7c7e4a12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:10.006120 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.006075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-dshm" (OuterVolumeSpecName: "dshm") pod "6c5751d3-515c-428b-9c76-cbdc7c7e4a12" (UID: "6c5751d3-515c-428b-9c76-cbdc7c7e4a12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103867    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103907    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m78r\" (UniqueName: \"kubernetes.io/projected/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-kube-api-access-6m78r\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103922    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mlsr\" (UniqueName: \"kubernetes.io/projected/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-kube-api-access-9mlsr\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103936    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103949    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103966    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103981    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.103992    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.104005    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.104020    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5751d3-515c-428b-9c76-cbdc7c7e4a12-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.107113 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.104032    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0217ee9c-e6ad-4b45-b748-3fda3e9cc791-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.109578 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.109538    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c5751d3-515c-428b-9c76-cbdc7c7e4a12" containerID="b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc" exitCode=0
Sep 12 03:06:10.109767 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.109744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2" event={"ID":"6c5751d3-515c-428b-9c76-cbdc7c7e4a12","Type":"ContainerDied","Data":"b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc"}
Sep 12 03:06:10.109859 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.109789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2" event={"ID":"6c5751d3-515c-428b-9c76-cbdc7c7e4a12","Type":"ContainerDied","Data":"4432f72e124a0c493e8e5e2dcfecba69d990b37bd4249c51797055a4b51cf560"}
Sep 12 03:06:10.109859 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.109820    2579 scope.go:117] "RemoveContainer" containerID="b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc"
Sep 12 03:06:10.110014 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.109999    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"
Sep 12 03:06:10.111940 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.111911    2579 generic.go:356] "Generic (PLEG): container finished" podID="0217ee9c-e6ad-4b45-b748-3fda3e9cc791" containerID="b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700" exitCode=0
Sep 12 03:06:10.112041 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.111954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7" event={"ID":"0217ee9c-e6ad-4b45-b748-3fda3e9cc791","Type":"ContainerDied","Data":"b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700"}
Sep 12 03:06:10.112041 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.111981    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"
Sep 12 03:06:10.112158 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.111984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7" event={"ID":"0217ee9c-e6ad-4b45-b748-3fda3e9cc791","Type":"ContainerDied","Data":"9f70a1fadd4c4aec46ae98507bfee647fbc19dcb81d6cdba2a0bd3a1c4459f04"}
Sep 12 03:06:10.119877 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.119750    2579 scope.go:117] "RemoveContainer" containerID="b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc"
Sep 12 03:06:10.120064 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:10.120044    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc\": container with ID starting with b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc not found: ID does not exist" containerID="b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc"
Sep 12 03:06:10.120163 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.120071    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc"} err="failed to get container status \"b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc\": rpc error: code = NotFound desc = could not find container \"b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc\": container with ID starting with b995e4854efcb868d30db40912f91e77d339de924345ea88748d6f4d21d4f1bc not found: ID does not exist"
Sep 12 03:06:10.120163 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.120118    2579 scope.go:117] "RemoveContainer" containerID="b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700"
Sep 12 03:06:10.128065 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.128049    2579 scope.go:117] "RemoveContainer" containerID="b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700"
Sep 12 03:06:10.128322 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:10.128305    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700\": container with ID starting with b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700 not found: ID does not exist" containerID="b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700"
Sep 12 03:06:10.128378 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.128330    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700"} err="failed to get container status \"b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700\": rpc error: code = NotFound desc = could not find container \"b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700\": container with ID starting with b00c8f3a8a50d194c1b5c05a00830fe84b4f138478bd3f1e0f5e5bd0b3da9700 not found: ID does not exist"
Sep 12 03:06:10.134661 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.134640    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"]
Sep 12 03:06:10.139614 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.139590    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/simulator-w4-c6190b-kserve-545658cb98-qscr7"]
Sep 12 03:06:10.148622 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.148602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"]
Sep 12 03:06:10.152036 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:10.152017    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/premium-w4-c6190b-kserve-fc8d8dcf9-g5rx2"]
Sep 12 03:06:11.249473 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:11.249438    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0217ee9c-e6ad-4b45-b748-3fda3e9cc791" path="/var/lib/kubelet/pods/0217ee9c-e6ad-4b45-b748-3fda3e9cc791/volumes"
Sep 12 03:06:11.249842 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:11.249832    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c5751d3-515c-428b-9c76-cbdc7c7e4a12" path="/var/lib/kubelet/pods/6c5751d3-515c-428b-9c76-cbdc7c7e4a12/volumes"
Sep 12 03:06:13.601296 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601263    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7882-5815f0"]
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601636    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451557bb-4bca-4342-b716-e1c95b08545c" containerName="maas-api"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601647    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="451557bb-4bca-4342-b716-e1c95b08545c" containerName="maas-api"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601657    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f99cad-7fd6-490f-b578-00d9742dde49" containerName="maas-api"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601662    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f99cad-7fd6-490f-b578-00d9742dde49" containerName="maas-api"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601671    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5751d3-515c-428b-9c76-cbdc7c7e4a12" containerName="main"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601677    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5751d3-515c-428b-9c76-cbdc7c7e4a12" containerName="main"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601685    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a213f8-3d1e-4670-8cec-56b5c0853758" containerName="main"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601691    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a213f8-3d1e-4670-8cec-56b5c0853758" containerName="main"
Sep 12 03:06:13.601696 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601697    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0217ee9c-e6ad-4b45-b748-3fda3e9cc791" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601703    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0217ee9c-e6ad-4b45-b748-3fda3e9cc791" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601709    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a71477-fd5b-47ed-9949-a30143d02142" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601714    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a71477-fd5b-47ed-9949-a30143d02142" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601729    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bac08d-5a46-48c4-b1e0-3b433ebf27ed" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601734    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bac08d-5a46-48c4-b1e0-3b433ebf27ed" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601742    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89c3a97-7694-47be-818f-eeb429d25743" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601747    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89c3a97-7694-47be-818f-eeb429d25743" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601754    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65a39b4-c76d-49ef-8a23-0a7649d1ae41" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601759    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65a39b4-c76d-49ef-8a23-0a7649d1ae41" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601812    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5751d3-515c-428b-9c76-cbdc7c7e4a12" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601818    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89c3a97-7694-47be-818f-eeb429d25743" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601823    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a71477-fd5b-47ed-9949-a30143d02142" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601828    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65a39b4-c76d-49ef-8a23-0a7649d1ae41" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601834    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a213f8-3d1e-4670-8cec-56b5c0853758" containerName="main"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601843    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bac08d-5a46-48c4-b1e0-3b433ebf27ed" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601848    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28f99cad-7fd6-490f-b578-00d9742dde49" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601854    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="451557bb-4bca-4342-b716-e1c95b08545c" containerName="maas-api"
Sep 12 03:06:13.601972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.601860    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0217ee9c-e6ad-4b45-b748-3fda3e9cc791" containerName="main"
Sep 12 03:06:13.606307 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.606284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:13.608816 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.608796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sww9\""
Sep 12 03:06:13.610839 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.610816    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7882-5815f0"]
Sep 12 03:06:13.738087 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.738051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjr9\" (UniqueName: \"kubernetes.io/projected/991c9128-93fc-4ed1-8704-0e11374659bb-kube-api-access-pmjr9\") pod \"test-curl-7882-5815f0\" (UID: \"991c9128-93fc-4ed1-8704-0e11374659bb\") " pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:13.838850 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.838812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjr9\" (UniqueName: \"kubernetes.io/projected/991c9128-93fc-4ed1-8704-0e11374659bb-kube-api-access-pmjr9\") pod \"test-curl-7882-5815f0\" (UID: \"991c9128-93fc-4ed1-8704-0e11374659bb\") " pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:13.846856 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.846834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjr9\" (UniqueName: \"kubernetes.io/projected/991c9128-93fc-4ed1-8704-0e11374659bb-kube-api-access-pmjr9\") pod \"test-curl-7882-5815f0\" (UID: \"991c9128-93fc-4ed1-8704-0e11374659bb\") " pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:13.917629 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:13.917563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:14.183042 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:14.183013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7882-5815f0"]
Sep 12 03:06:14.185496 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:06:14.185472    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod991c9128_93fc_4ed1_8704_0e11374659bb.slice/crio-89a29d3bc611f87f07dde6097bf41eeb6fbf8609734fe4d2d9c6c0c5fac0c52d WatchSource:0}: Error finding container 89a29d3bc611f87f07dde6097bf41eeb6fbf8609734fe4d2d9c6c0c5fac0c52d: Status 404 returned error can't find the container with id 89a29d3bc611f87f07dde6097bf41eeb6fbf8609734fe4d2d9c6c0c5fac0c52d
Sep 12 03:06:15.141852 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:15.141818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7882-5815f0" event={"ID":"991c9128-93fc-4ed1-8704-0e11374659bb","Type":"ContainerStarted","Data":"1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff"}
Sep 12 03:06:15.142261 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:15.141858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7882-5815f0" event={"ID":"991c9128-93fc-4ed1-8704-0e11374659bb","Type":"ContainerStarted","Data":"89a29d3bc611f87f07dde6097bf41eeb6fbf8609734fe4d2d9c6c0c5fac0c52d"}
Sep 12 03:06:15.155423 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:15.155379    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7882-5815f0" podStartSLOduration=2.155362151 podStartE2EDuration="2.155362151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:06:15.154593701 +0000 UTC m=+1814.422627347" watchObservedRunningTime="2026-09-12 03:06:15.155362151 +0000 UTC m=+1814.423395800"
Sep 12 03:06:15.753010 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:15.752974    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7882-5815f0"]
Sep 12 03:06:15.755001 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:15.754975    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7882-5815f0"]
Sep 12 03:06:16.146832 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:16.146797    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7882-5815f0" container="test-curl-7882-5815f0"
Sep 12 03:06:17.151566 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:17.151524    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7882-5815f0" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" containerName="test-curl-7882-5815f0" containerID="cri-o://1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff" gracePeriod=2
Sep 12 03:06:17.153874 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:17.153842    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7882-5815f0\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:19.336640 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:19.336618    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:19.498678 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:19.498631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmjr9\" (UniqueName: \"kubernetes.io/projected/991c9128-93fc-4ed1-8704-0e11374659bb-kube-api-access-pmjr9\") pod \"991c9128-93fc-4ed1-8704-0e11374659bb\" (UID: \"991c9128-93fc-4ed1-8704-0e11374659bb\") "
Sep 12 03:06:19.500698 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:19.500670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991c9128-93fc-4ed1-8704-0e11374659bb-kube-api-access-pmjr9" (OuterVolumeSpecName: "kube-api-access-pmjr9") pod "991c9128-93fc-4ed1-8704-0e11374659bb" (UID: "991c9128-93fc-4ed1-8704-0e11374659bb"). InnerVolumeSpecName "kube-api-access-pmjr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:19.599837 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:19.599803    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjr9\" (UniqueName: \"kubernetes.io/projected/991c9128-93fc-4ed1-8704-0e11374659bb-kube-api-access-pmjr9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:19.601877 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:19.601775    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"]
Sep 12 03:06:20.168230 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:20.168198    2579 generic.go:356] "Generic (PLEG): container finished" podID="991c9128-93fc-4ed1-8704-0e11374659bb" containerID="1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff" exitCode=137
Sep 12 03:06:20.168381 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:20.168241    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:20.168381 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:20.168299    2579 scope.go:117] "RemoveContainer" containerID="1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff"
Sep 12 03:06:20.177043 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:20.177025    2579 scope.go:117] "RemoveContainer" containerID="1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff"
Sep 12 03:06:20.177342 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:20.177323    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff\": container with ID starting with 1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff not found: ID does not exist" containerID="1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff"
Sep 12 03:06:20.177418 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:20.177349    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff"} err="failed to get container status \"1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff\": rpc error: code = NotFound desc = could not find container \"1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff\": container with ID starting with 1200bc85e4e5fb79ff4a484e071796a7c8956067d0837eccb2d3b4edb03303ff not found: ID does not exist"
Sep 12 03:06:20.178034 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:20.178014    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7882-5815f0\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:20.240457 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:20.240431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:06:21.247590 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:21.247557    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7882-5815f0\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" pod="opendatahub/test-curl-7882-5815f0"
Sep 12 03:06:21.249102 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:21.249075    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" path="/var/lib/kubelet/pods/991c9128-93fc-4ed1-8704-0e11374659bb/volumes"
Sep 12 03:06:22.899639 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:22.899604    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"]
Sep 12 03:06:31.948501 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:31.948467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:06:34.621636 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.621577    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8" podUID="c35849b3-dedd-4637-9c1c-fb90700198bb" containerName="main" containerID="cri-o://c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf" gracePeriod=45
Sep 12 03:06:34.869642 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.869620    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:06:34.939657 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.939621    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35849b3-dedd-4637-9c1c-fb90700198bb-tls-certs\") pod \"c35849b3-dedd-4637-9c1c-fb90700198bb\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") "
Sep 12 03:06:34.939842 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.939727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-home\") pod \"c35849b3-dedd-4637-9c1c-fb90700198bb\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") "
Sep 12 03:06:34.939842 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.939789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-dshm\") pod \"c35849b3-dedd-4637-9c1c-fb90700198bb\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") "
Sep 12 03:06:34.939842 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.939830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpng6\" (UniqueName: \"kubernetes.io/projected/c35849b3-dedd-4637-9c1c-fb90700198bb-kube-api-access-zpng6\") pod \"c35849b3-dedd-4637-9c1c-fb90700198bb\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") "
Sep 12 03:06:34.940010 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.939862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-model-cache\") pod \"c35849b3-dedd-4637-9c1c-fb90700198bb\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") "
Sep 12 03:06:34.940010 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.939888    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-tmp-dir\") pod \"c35849b3-dedd-4637-9c1c-fb90700198bb\" (UID: \"c35849b3-dedd-4637-9c1c-fb90700198bb\") "
Sep 12 03:06:34.940130 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.940002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-home" (OuterVolumeSpecName: "home") pod "c35849b3-dedd-4637-9c1c-fb90700198bb" (UID: "c35849b3-dedd-4637-9c1c-fb90700198bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:34.940186 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.940143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c35849b3-dedd-4637-9c1c-fb90700198bb" (UID: "c35849b3-dedd-4637-9c1c-fb90700198bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:34.940247 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.940212    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:34.940247 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.940211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c35849b3-dedd-4637-9c1c-fb90700198bb" (UID: "c35849b3-dedd-4637-9c1c-fb90700198bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:34.942048 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.942027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35849b3-dedd-4637-9c1c-fb90700198bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c35849b3-dedd-4637-9c1c-fb90700198bb" (UID: "c35849b3-dedd-4637-9c1c-fb90700198bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:34.942267 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.942242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-dshm" (OuterVolumeSpecName: "dshm") pod "c35849b3-dedd-4637-9c1c-fb90700198bb" (UID: "c35849b3-dedd-4637-9c1c-fb90700198bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:34.942377 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:34.942269    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35849b3-dedd-4637-9c1c-fb90700198bb-kube-api-access-zpng6" (OuterVolumeSpecName: "kube-api-access-zpng6") pod "c35849b3-dedd-4637-9c1c-fb90700198bb" (UID: "c35849b3-dedd-4637-9c1c-fb90700198bb"). InnerVolumeSpecName "kube-api-access-zpng6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:35.040900 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.040855    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:35.040900 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.040898    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpng6\" (UniqueName: \"kubernetes.io/projected/c35849b3-dedd-4637-9c1c-fb90700198bb-kube-api-access-zpng6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:35.040900 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.040909    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:35.041171 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.040938    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c35849b3-dedd-4637-9c1c-fb90700198bb-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:35.041171 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.040945    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c35849b3-dedd-4637-9c1c-fb90700198bb-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:35.253809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.253727    2579 generic.go:356] "Generic (PLEG): container finished" podID="c35849b3-dedd-4637-9c1c-fb90700198bb" containerID="c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf" exitCode=0
Sep 12 03:06:35.253809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.253769    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"
Sep 12 03:06:35.253809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.253780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8" event={"ID":"c35849b3-dedd-4637-9c1c-fb90700198bb","Type":"ContainerDied","Data":"c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf"}
Sep 12 03:06:35.253809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.253810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8" event={"ID":"c35849b3-dedd-4637-9c1c-fb90700198bb","Type":"ContainerDied","Data":"ea96a362309c93946aa20609cc2298a6ccb114a8a2c4a972db5e09679192b5fc"}
Sep 12 03:06:35.254118 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.253827    2579 scope.go:117] "RemoveContainer" containerID="c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf"
Sep 12 03:06:35.263360 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.263342    2579 scope.go:117] "RemoveContainer" containerID="c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf"
Sep 12 03:06:35.263620 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:35.263601    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf\": container with ID starting with c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf not found: ID does not exist" containerID="c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf"
Sep 12 03:06:35.263671 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.263629    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf"} err="failed to get container status \"c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf\": rpc error: code = NotFound desc = could not find container \"c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf\": container with ID starting with c2e1d10b4e077736168ad1fa015bfc0780a5647018b60ebffc84df16d447f2bf not found: ID does not exist"
Sep 12 03:06:35.275461 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.275436    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"]
Sep 12 03:06:35.280231 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:35.280210    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/auth-test-model-f91169-kserve-554975595-dsxm8"]
Sep 12 03:06:37.249089 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:37.249048    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35849b3-dedd-4637-9c1c-fb90700198bb" path="/var/lib/kubelet/pods/c35849b3-dedd-4637-9c1c-fb90700198bb/volumes"
Sep 12 03:06:37.920069 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:37.920021    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6" podUID="0488848d-2317-4f70-9237-5ae16daf1fbb" containerName="main" containerID="cri-o://775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d" gracePeriod=45
Sep 12 03:06:38.173924 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.173860    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:06:38.272219 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-model-cache\") pod \"0488848d-2317-4f70-9237-5ae16daf1fbb\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") "
Sep 12 03:06:38.272699 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0488848d-2317-4f70-9237-5ae16daf1fbb-tls-certs\") pod \"0488848d-2317-4f70-9237-5ae16daf1fbb\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") "
Sep 12 03:06:38.272699 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-dshm\") pod \"0488848d-2317-4f70-9237-5ae16daf1fbb\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") "
Sep 12 03:06:38.272699 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272349    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-home\") pod \"0488848d-2317-4f70-9237-5ae16daf1fbb\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") "
Sep 12 03:06:38.272699 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf6bv\" (UniqueName: \"kubernetes.io/projected/0488848d-2317-4f70-9237-5ae16daf1fbb-kube-api-access-qf6bv\") pod \"0488848d-2317-4f70-9237-5ae16daf1fbb\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") "
Sep 12 03:06:38.272699 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-tmp-dir\") pod \"0488848d-2317-4f70-9237-5ae16daf1fbb\" (UID: \"0488848d-2317-4f70-9237-5ae16daf1fbb\") "
Sep 12 03:06:38.272699 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "0488848d-2317-4f70-9237-5ae16daf1fbb" (UID: "0488848d-2317-4f70-9237-5ae16daf1fbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:38.272699 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272688    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-home" (OuterVolumeSpecName: "home") pod "0488848d-2317-4f70-9237-5ae16daf1fbb" (UID: "0488848d-2317-4f70-9237-5ae16daf1fbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:38.273119 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272736    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:38.273119 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.272933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0488848d-2317-4f70-9237-5ae16daf1fbb" (UID: "0488848d-2317-4f70-9237-5ae16daf1fbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:38.274978 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.274954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0488848d-2317-4f70-9237-5ae16daf1fbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0488848d-2317-4f70-9237-5ae16daf1fbb" (UID: "0488848d-2317-4f70-9237-5ae16daf1fbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:38.275126 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.275029    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0488848d-2317-4f70-9237-5ae16daf1fbb-kube-api-access-qf6bv" (OuterVolumeSpecName: "kube-api-access-qf6bv") pod "0488848d-2317-4f70-9237-5ae16daf1fbb" (UID: "0488848d-2317-4f70-9237-5ae16daf1fbb"). InnerVolumeSpecName "kube-api-access-qf6bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:38.275126 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.275115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-dshm" (OuterVolumeSpecName: "dshm") pod "0488848d-2317-4f70-9237-5ae16daf1fbb" (UID: "0488848d-2317-4f70-9237-5ae16daf1fbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:38.275220 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.275191    2579 generic.go:356] "Generic (PLEG): container finished" podID="0488848d-2317-4f70-9237-5ae16daf1fbb" containerID="775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d" exitCode=0
Sep 12 03:06:38.275254 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.275232    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"
Sep 12 03:06:38.275286 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.275250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6" event={"ID":"0488848d-2317-4f70-9237-5ae16daf1fbb","Type":"ContainerDied","Data":"775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d"}
Sep 12 03:06:38.275382 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.275291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6" event={"ID":"0488848d-2317-4f70-9237-5ae16daf1fbb","Type":"ContainerDied","Data":"b4fd71389b490f518073b87967207ce29eff943109e4060ed3f254a8292daeee"}
Sep 12 03:06:38.275382 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.275314    2579 scope.go:117] "RemoveContainer" containerID="775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d"
Sep 12 03:06:38.295361 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.295339    2579 scope.go:117] "RemoveContainer" containerID="775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d"
Sep 12 03:06:38.295709 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:06:38.295688    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d\": container with ID starting with 775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d not found: ID does not exist" containerID="775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d"
Sep 12 03:06:38.295766 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.295720    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d"} err="failed to get container status \"775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d\": rpc error: code = NotFound desc = could not find container \"775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d\": container with ID starting with 775982ed09550ba03aa78d53a1fa6b160599859492982e26fb023fcc0f615d2d not found: ID does not exist"
Sep 12 03:06:38.305150 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.305128    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"]
Sep 12 03:06:38.308656 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.308632    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/auth-test-model-e401bd-kserve-7db746f5-l5gk6"]
Sep 12 03:06:38.373590 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.373562    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0488848d-2317-4f70-9237-5ae16daf1fbb-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:38.373590 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.373587    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:38.373590 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.373595    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:38.373803 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.373603    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf6bv\" (UniqueName: \"kubernetes.io/projected/0488848d-2317-4f70-9237-5ae16daf1fbb-kube-api-access-qf6bv\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:38.373803 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:38.373612    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0488848d-2317-4f70-9237-5ae16daf1fbb-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:06:39.248620 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:39.248586    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0488848d-2317-4f70-9237-5ae16daf1fbb" path="/var/lib/kubelet/pods/0488848d-2317-4f70-9237-5ae16daf1fbb/volumes"
Sep 12 03:06:43.342423 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:06:43.342391    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:07:01.468618 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:01.468586    2579 scope.go:117] "RemoveContainer" containerID="e2a6861dcf2ccf65b27353da2083256561c819e37e2133753cbaa190db9d723f"
Sep 12 03:07:10.425374 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.425331    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"]
Sep 12 03:07:10.425952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.425929    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35849b3-dedd-4637-9c1c-fb90700198bb" containerName="main"
Sep 12 03:07:10.426041 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.425955    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35849b3-dedd-4637-9c1c-fb90700198bb" containerName="main"
Sep 12 03:07:10.426041 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.425971    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0488848d-2317-4f70-9237-5ae16daf1fbb" containerName="main"
Sep 12 03:07:10.426041 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.425981    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0488848d-2317-4f70-9237-5ae16daf1fbb" containerName="main"
Sep 12 03:07:10.426041 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.426015    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" containerName="test-curl-7882-5815f0"
Sep 12 03:07:10.426041 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.426025    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" containerName="test-curl-7882-5815f0"
Sep 12 03:07:10.426324 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.426164    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0488848d-2317-4f70-9237-5ae16daf1fbb" containerName="main"
Sep 12 03:07:10.426324 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.426182    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35849b3-dedd-4637-9c1c-fb90700198bb" containerName="main"
Sep 12 03:07:10.426324 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.426194    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="991c9128-93fc-4ed1-8704-0e11374659bb" containerName="test-curl-7882-5815f0"
Sep 12 03:07:10.429367 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.429341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.431961 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.431943    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"openshift-service-ca.crt\""
Sep 12 03:07:10.433027 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.433003    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"kube-root-ca.crt\""
Sep 12 03:07:10.433144 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.433064    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"default-dockercfg-f52m4\""
Sep 12 03:07:10.433222 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.433200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"sub-test-model-e401bd-kserve-self-signed-certs\""
Sep 12 03:07:10.438353 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.438329    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"]
Sep 12 03:07:10.457642 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.457611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-dshm\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.457762 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.457660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/016789a7-277b-4108-8648-407ff0784119-tls-certs\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.457762 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.457714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-model-cache\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.457762 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.457754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-tmp-dir\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.457910 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.457846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts57f\" (UniqueName: \"kubernetes.io/projected/016789a7-277b-4108-8648-407ff0784119-kube-api-access-ts57f\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.457910 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.457871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-home\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.558935 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.558901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts57f\" (UniqueName: \"kubernetes.io/projected/016789a7-277b-4108-8648-407ff0784119-kube-api-access-ts57f\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.558935 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.558939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-home\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.559193 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.558968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-dshm\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.559193 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.558992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/016789a7-277b-4108-8648-407ff0784119-tls-certs\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.559193 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.559037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-model-cache\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.559193 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.559059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-tmp-dir\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.559397 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.559376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-home\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.559455 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.559424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-tmp-dir\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.559539 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.559517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-model-cache\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.561279 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.561258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-dshm\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.561556 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.561540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/016789a7-277b-4108-8648-407ff0784119-tls-certs\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.567014 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.566990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts57f\" (UniqueName: \"kubernetes.io/projected/016789a7-277b-4108-8648-407ff0784119-kube-api-access-ts57f\") pod \"sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:10.739088 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:10.739055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:11.008756 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:11.008685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"]
Sep 12 03:07:11.011882 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:07:11.011843    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod016789a7_277b_4108_8648_407ff0784119.slice/crio-794deda501b584d8c80fc7c0b9e25b987d62830a63b00103486a6dfc86bd3b10 WatchSource:0}: Error finding container 794deda501b584d8c80fc7c0b9e25b987d62830a63b00103486a6dfc86bd3b10: Status 404 returned error can't find the container with id 794deda501b584d8c80fc7c0b9e25b987d62830a63b00103486a6dfc86bd3b10
Sep 12 03:07:11.436902 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:11.436872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r" event={"ID":"016789a7-277b-4108-8648-407ff0784119","Type":"ContainerStarted","Data":"239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56"}
Sep 12 03:07:11.437309 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:11.436915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r" event={"ID":"016789a7-277b-4108-8648-407ff0784119","Type":"ContainerStarted","Data":"794deda501b584d8c80fc7c0b9e25b987d62830a63b00103486a6dfc86bd3b10"}
Sep 12 03:07:11.455154 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:11.455085    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r" podStartSLOduration=1.455071765 podStartE2EDuration="1.455071765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:07:11.452889322 +0000 UTC m=+1870.720922972" watchObservedRunningTime="2026-09-12 03:07:11.455071765 +0000 UTC m=+1870.723105412"
Sep 12 03:07:14.053740 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:14.053703    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:07:20.740271 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:20.740224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:20.740675 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:20.740292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:20.752616 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:20.752591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:21.498725 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:21.498691    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:07:31.839814 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:31.839736    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:07:42.346011 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.345976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7882-451856"]
Sep 12 03:07:42.349659 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.349642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:42.352044 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.352020    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sww9\""
Sep 12 03:07:42.355261 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.355242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7882-451856"]
Sep 12 03:07:42.459720 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.459683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktwt\" (UniqueName: \"kubernetes.io/projected/401edeac-8997-4126-8cc8-0d775e111cba-kube-api-access-hktwt\") pod \"test-curl-7882-451856\" (UID: \"401edeac-8997-4126-8cc8-0d775e111cba\") " pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:42.560566 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.560539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktwt\" (UniqueName: \"kubernetes.io/projected/401edeac-8997-4126-8cc8-0d775e111cba-kube-api-access-hktwt\") pod \"test-curl-7882-451856\" (UID: \"401edeac-8997-4126-8cc8-0d775e111cba\") " pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:42.570347 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.570312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktwt\" (UniqueName: \"kubernetes.io/projected/401edeac-8997-4126-8cc8-0d775e111cba-kube-api-access-hktwt\") pod \"test-curl-7882-451856\" (UID: \"401edeac-8997-4126-8cc8-0d775e111cba\") " pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:42.660457 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.660392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:42.928370 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:42.928349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7882-451856"]
Sep 12 03:07:42.930531 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:07:42.930496    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod401edeac_8997_4126_8cc8_0d775e111cba.slice/crio-ee37a0f8870d4e1d77e9abfd6e103936877d508a8a982db7bc5c81ada6e2b6a4 WatchSource:0}: Error finding container ee37a0f8870d4e1d77e9abfd6e103936877d508a8a982db7bc5c81ada6e2b6a4: Status 404 returned error can't find the container with id ee37a0f8870d4e1d77e9abfd6e103936877d508a8a982db7bc5c81ada6e2b6a4
Sep 12 03:07:43.602513 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:43.602473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7882-451856" event={"ID":"401edeac-8997-4126-8cc8-0d775e111cba","Type":"ContainerStarted","Data":"2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e"}
Sep 12 03:07:43.602891 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:43.602528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7882-451856" event={"ID":"401edeac-8997-4126-8cc8-0d775e111cba","Type":"ContainerStarted","Data":"ee37a0f8870d4e1d77e9abfd6e103936877d508a8a982db7bc5c81ada6e2b6a4"}
Sep 12 03:07:43.617500 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:43.617448    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7882-451856" podStartSLOduration=1.6174312469999998 podStartE2EDuration="1.617431247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:07:43.615684791 +0000 UTC m=+1902.883718442" watchObservedRunningTime="2026-09-12 03:07:43.617431247 +0000 UTC m=+1902.885464893"
Sep 12 03:07:44.218054 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.218019    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7882-451856"]
Sep 12 03:07:44.219969 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.219944    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7882-451856"]
Sep 12 03:07:44.342620 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.342543    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7882-dc0769"]
Sep 12 03:07:44.342955 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.342942    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401edeac-8997-4126-8cc8-0d775e111cba" containerName="test-curl-7882-451856"
Sep 12 03:07:44.343001 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.342957    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="401edeac-8997-4126-8cc8-0d775e111cba" containerName="test-curl-7882-451856"
Sep 12 03:07:44.343034 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.343014    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="401edeac-8997-4126-8cc8-0d775e111cba" containerName="test-curl-7882-451856"
Sep 12 03:07:44.345616 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.345601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:44.351682 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.351648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7882-dc0769"]
Sep 12 03:07:44.479187 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.479157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5m9\" (UniqueName: \"kubernetes.io/projected/02b3954a-fa05-4f3c-b3a7-42db662fef13-kube-api-access-jd5m9\") pod \"test-curl-7882-dc0769\" (UID: \"02b3954a-fa05-4f3c-b3a7-42db662fef13\") " pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:44.580472 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.580441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5m9\" (UniqueName: \"kubernetes.io/projected/02b3954a-fa05-4f3c-b3a7-42db662fef13-kube-api-access-jd5m9\") pod \"test-curl-7882-dc0769\" (UID: \"02b3954a-fa05-4f3c-b3a7-42db662fef13\") " pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:44.588542 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.588523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5m9\" (UniqueName: \"kubernetes.io/projected/02b3954a-fa05-4f3c-b3a7-42db662fef13-kube-api-access-jd5m9\") pod \"test-curl-7882-dc0769\" (UID: \"02b3954a-fa05-4f3c-b3a7-42db662fef13\") " pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:44.607640 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:07:44.607585    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7882-451856" container="test-curl-7882-451856"
Sep 12 03:07:44.656633 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.656609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:44.925161 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:44.925037    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7882-dc0769"]
Sep 12 03:07:44.927273 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:07:44.927243    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b3954a_fa05_4f3c_b3a7_42db662fef13.slice/crio-b230d845144dba6ac63bfe3ca4d95eebbd8cb5a254388dc901ae753a1a3a517c WatchSource:0}: Error finding container b230d845144dba6ac63bfe3ca4d95eebbd8cb5a254388dc901ae753a1a3a517c: Status 404 returned error can't find the container with id b230d845144dba6ac63bfe3ca4d95eebbd8cb5a254388dc901ae753a1a3a517c
Sep 12 03:07:45.613687 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:45.613649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7882-dc0769" event={"ID":"02b3954a-fa05-4f3c-b3a7-42db662fef13","Type":"ContainerStarted","Data":"943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973"}
Sep 12 03:07:45.614157 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:45.613700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7882-dc0769" event={"ID":"02b3954a-fa05-4f3c-b3a7-42db662fef13","Type":"ContainerStarted","Data":"b230d845144dba6ac63bfe3ca4d95eebbd8cb5a254388dc901ae753a1a3a517c"}
Sep 12 03:07:45.614157 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:45.613720    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7882-451856" podUID="401edeac-8997-4126-8cc8-0d775e111cba" containerName="test-curl-7882-451856" containerID="cri-o://2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e" gracePeriod=2
Sep 12 03:07:45.628682 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:45.628624    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7882-dc0769" podStartSLOduration=1.628604674 podStartE2EDuration="1.628604674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:07:45.625722175 +0000 UTC m=+1904.893755821" watchObservedRunningTime="2026-09-12 03:07:45.628604674 +0000 UTC m=+1904.896638322"
Sep 12 03:07:46.227685 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:46.227649    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7882-dc0769"]
Sep 12 03:07:46.231118 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:46.231076    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7882-dc0769"]
Sep 12 03:07:46.617929 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:07:46.617894    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7882-dc0769" container="test-curl-7882-dc0769"
Sep 12 03:07:47.439908 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:47.439871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:07:47.622919 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:47.622887    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7882-dc0769" podUID="02b3954a-fa05-4f3c-b3a7-42db662fef13" containerName="test-curl-7882-dc0769" containerID="cri-o://943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973" gracePeriod=2
Sep 12 03:07:47.787540 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:47.787515    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:47.911250 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:47.911228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hktwt\" (UniqueName: \"kubernetes.io/projected/401edeac-8997-4126-8cc8-0d775e111cba-kube-api-access-hktwt\") pod \"401edeac-8997-4126-8cc8-0d775e111cba\" (UID: \"401edeac-8997-4126-8cc8-0d775e111cba\") "
Sep 12 03:07:47.913419 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:47.913391    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401edeac-8997-4126-8cc8-0d775e111cba-kube-api-access-hktwt" (OuterVolumeSpecName: "kube-api-access-hktwt") pod "401edeac-8997-4126-8cc8-0d775e111cba" (UID: "401edeac-8997-4126-8cc8-0d775e111cba"). InnerVolumeSpecName "kube-api-access-hktwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:07:48.012030 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:48.011949    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktwt\" (UniqueName: \"kubernetes.io/projected/401edeac-8997-4126-8cc8-0d775e111cba-kube-api-access-hktwt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:07:48.629040 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:48.629004    2579 generic.go:356] "Generic (PLEG): container finished" podID="401edeac-8997-4126-8cc8-0d775e111cba" containerID="2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e" exitCode=137
Sep 12 03:07:48.629454 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:48.629047    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:48.629454 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:48.629110    2579 scope.go:117] "RemoveContainer" containerID="2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e"
Sep 12 03:07:48.638086 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:48.638070    2579 scope.go:117] "RemoveContainer" containerID="2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e"
Sep 12 03:07:48.638384 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:07:48.638367    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e\": container with ID starting with 2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e not found: ID does not exist" containerID="2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e"
Sep 12 03:07:48.638448 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:48.638394    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e"} err="failed to get container status \"2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e\": rpc error: code = NotFound desc = could not find container \"2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e\": container with ID starting with 2d702634d912a163e46bb4eda355fac15a69df3be3373d57790f212378e9be8e not found: ID does not exist"
Sep 12 03:07:48.639061 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:07:48.639037    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7882-451856\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="401edeac-8997-4126-8cc8-0d775e111cba" pod="opendatahub/test-curl-7882-451856"
Sep 12 03:07:49.249662 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:49.249631    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401edeac-8997-4126-8cc8-0d775e111cba" path="/var/lib/kubelet/pods/401edeac-8997-4126-8cc8-0d775e111cba/volumes"
Sep 12 03:07:49.403896 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:49.403863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"]
Sep 12 03:07:49.816178 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:49.816153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:49.929525 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:49.929498    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd5m9\" (UniqueName: \"kubernetes.io/projected/02b3954a-fa05-4f3c-b3a7-42db662fef13-kube-api-access-jd5m9\") pod \"02b3954a-fa05-4f3c-b3a7-42db662fef13\" (UID: \"02b3954a-fa05-4f3c-b3a7-42db662fef13\") "
Sep 12 03:07:49.931690 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:49.931666    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b3954a-fa05-4f3c-b3a7-42db662fef13-kube-api-access-jd5m9" (OuterVolumeSpecName: "kube-api-access-jd5m9") pod "02b3954a-fa05-4f3c-b3a7-42db662fef13" (UID: "02b3954a-fa05-4f3c-b3a7-42db662fef13"). InnerVolumeSpecName "kube-api-access-jd5m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:07:50.030676 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:50.030649    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd5m9\" (UniqueName: \"kubernetes.io/projected/02b3954a-fa05-4f3c-b3a7-42db662fef13-kube-api-access-jd5m9\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:07:50.641880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:50.641844    2579 generic.go:356] "Generic (PLEG): container finished" podID="02b3954a-fa05-4f3c-b3a7-42db662fef13" containerID="943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973" exitCode=137
Sep 12 03:07:50.641880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:50.641884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:50.642132 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:50.641950    2579 scope.go:117] "RemoveContainer" containerID="943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973"
Sep 12 03:07:50.651182 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:50.651165    2579 scope.go:117] "RemoveContainer" containerID="943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973"
Sep 12 03:07:50.651429 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:07:50.651412    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973\": container with ID starting with 943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973 not found: ID does not exist" containerID="943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973"
Sep 12 03:07:50.651472 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:50.651437    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973"} err="failed to get container status \"943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973\": rpc error: code = NotFound desc = could not find container \"943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973\": container with ID starting with 943fea8537781a539f8d0aa6cbd6bd79e8437778db8331ccef7fbd175256c973 not found: ID does not exist"
Sep 12 03:07:50.651744 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:07:50.651722    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7882-dc0769\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="02b3954a-fa05-4f3c-b3a7-42db662fef13" pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:51.247269 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:07:51.247232    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7882-dc0769\" is forbidden: User \"system:node:ip-10-0-130-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-57.ec2.internal' and this object" podUID="02b3954a-fa05-4f3c-b3a7-42db662fef13" pod="opendatahub/test-curl-7882-dc0769"
Sep 12 03:07:51.249184 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:51.249160    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b3954a-fa05-4f3c-b3a7-42db662fef13" path="/var/lib/kubelet/pods/02b3954a-fa05-4f3c-b3a7-42db662fef13/volumes"
Sep 12 03:07:58.052693 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:07:58.052653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:08:03.121779 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.121748    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"]
Sep 12 03:08:03.122338 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.122319    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b3954a-fa05-4f3c-b3a7-42db662fef13" containerName="test-curl-7882-dc0769"
Sep 12 03:08:03.122416 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.122342    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b3954a-fa05-4f3c-b3a7-42db662fef13" containerName="test-curl-7882-dc0769"
Sep 12 03:08:03.122480 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.122469    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b3954a-fa05-4f3c-b3a7-42db662fef13" containerName="test-curl-7882-dc0769"
Sep 12 03:08:03.125511 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.125491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.129365 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.129331    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"rate-test-model-f91169-kserve-self-signed-certs\""
Sep 12 03:08:03.129365 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.129348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"kube-root-ca.crt\""
Sep 12 03:08:03.129365 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.129338    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"default-dockercfg-pc6s2\""
Sep 12 03:08:03.129597 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.129355    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"openshift-service-ca.crt\""
Sep 12 03:08:03.136584 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.136561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"]
Sep 12 03:08:03.249594 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.249565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-tmp-dir\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.249764 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.249605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463845f6-f53b-453b-9a76-6d9d21616d82-tls-certs\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.249764 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.249630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-model-cache\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.249859 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.249745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-dshm\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.249859 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.249817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44r7t\" (UniqueName: \"kubernetes.io/projected/463845f6-f53b-453b-9a76-6d9d21616d82-kube-api-access-44r7t\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.249951 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.249855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-home\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351172 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-home\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351345 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-tmp-dir\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351345 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463845f6-f53b-453b-9a76-6d9d21616d82-tls-certs\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351345 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-model-cache\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351345 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-dshm\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351345 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44r7t\" (UniqueName: \"kubernetes.io/projected/463845f6-f53b-453b-9a76-6d9d21616d82-kube-api-access-44r7t\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351629 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-home\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351673 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-model-cache\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.351717 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.351679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-tmp-dir\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.353629 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.353607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-dshm\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.353893 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.353877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463845f6-f53b-453b-9a76-6d9d21616d82-tls-certs\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.366948 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.366926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44r7t\" (UniqueName: \"kubernetes.io/projected/463845f6-f53b-453b-9a76-6d9d21616d82-kube-api-access-44r7t\") pod \"rate-test-model-f91169-kserve-6599d9bc55-hh924\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") " pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.436088 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.436062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:03.707167 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:03.707146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"]
Sep 12 03:08:03.709178 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:08:03.709145    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod463845f6_f53b_453b_9a76_6d9d21616d82.slice/crio-7713d7e5a8b852f4a8d5d76a02c7efb5a5ec2a3d9d000f0d7c5d7336d89cdc52 WatchSource:0}: Error finding container 7713d7e5a8b852f4a8d5d76a02c7efb5a5ec2a3d9d000f0d7c5d7336d89cdc52: Status 404 returned error can't find the container with id 7713d7e5a8b852f4a8d5d76a02c7efb5a5ec2a3d9d000f0d7c5d7336d89cdc52
Sep 12 03:08:04.423426 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.423389    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r" podUID="016789a7-277b-4108-8648-407ff0784119" containerName="main" containerID="cri-o://239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56" gracePeriod=45
Sep 12 03:08:04.684085 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.684061    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:08:04.714376 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.714344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924" event={"ID":"463845f6-f53b-453b-9a76-6d9d21616d82","Type":"ContainerStarted","Data":"749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea"}
Sep 12 03:08:04.714516 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.714389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924" event={"ID":"463845f6-f53b-453b-9a76-6d9d21616d82","Type":"ContainerStarted","Data":"7713d7e5a8b852f4a8d5d76a02c7efb5a5ec2a3d9d000f0d7c5d7336d89cdc52"}
Sep 12 03:08:04.715693 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.715659    2579 generic.go:356] "Generic (PLEG): container finished" podID="016789a7-277b-4108-8648-407ff0784119" containerID="239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56" exitCode=0
Sep 12 03:08:04.715693 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.715694    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"
Sep 12 03:08:04.715838 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.715746    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r" event={"ID":"016789a7-277b-4108-8648-407ff0784119","Type":"ContainerDied","Data":"239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56"}
Sep 12 03:08:04.715838 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.715795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r" event={"ID":"016789a7-277b-4108-8648-407ff0784119","Type":"ContainerDied","Data":"794deda501b584d8c80fc7c0b9e25b987d62830a63b00103486a6dfc86bd3b10"}
Sep 12 03:08:04.715906 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.715819    2579 scope.go:117] "RemoveContainer" containerID="239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56"
Sep 12 03:08:04.724790 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.724776    2579 scope.go:117] "RemoveContainer" containerID="239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56"
Sep 12 03:08:04.725046 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:08:04.725026    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56\": container with ID starting with 239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56 not found: ID does not exist" containerID="239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56"
Sep 12 03:08:04.725135 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.725055    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56"} err="failed to get container status \"239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56\": rpc error: code = NotFound desc = could not find container \"239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56\": container with ID starting with 239960e2cddbeba1d8d976ff1983aa1b12d2c930264ca0d1092aaf5ed56ceb56 not found: ID does not exist"
Sep 12 03:08:04.732496 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.732454    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924" podStartSLOduration=1.732440207 podStartE2EDuration="1.732440207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:08:04.731988437 +0000 UTC m=+1924.000022083" watchObservedRunningTime="2026-09-12 03:08:04.732440207 +0000 UTC m=+1924.000473853"
Sep 12 03:08:04.764693 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.764674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-model-cache\") pod \"016789a7-277b-4108-8648-407ff0784119\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") "
Sep 12 03:08:04.764807 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.764703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-dshm\") pod \"016789a7-277b-4108-8648-407ff0784119\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") "
Sep 12 03:08:04.764807 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.764730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-tmp-dir\") pod \"016789a7-277b-4108-8648-407ff0784119\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") "
Sep 12 03:08:04.764807 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.764792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts57f\" (UniqueName: \"kubernetes.io/projected/016789a7-277b-4108-8648-407ff0784119-kube-api-access-ts57f\") pod \"016789a7-277b-4108-8648-407ff0784119\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") "
Sep 12 03:08:04.764983 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.764816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/016789a7-277b-4108-8648-407ff0784119-tls-certs\") pod \"016789a7-277b-4108-8648-407ff0784119\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") "
Sep 12 03:08:04.764983 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.764837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-home\") pod \"016789a7-277b-4108-8648-407ff0784119\" (UID: \"016789a7-277b-4108-8648-407ff0784119\") "
Sep 12 03:08:04.764983 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.764957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-model-cache" (OuterVolumeSpecName: "model-cache") pod "016789a7-277b-4108-8648-407ff0784119" (UID: "016789a7-277b-4108-8648-407ff0784119"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:04.765166 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.765055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "016789a7-277b-4108-8648-407ff0784119" (UID: "016789a7-277b-4108-8648-407ff0784119"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:04.765166 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.765151    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-home" (OuterVolumeSpecName: "home") pod "016789a7-277b-4108-8648-407ff0784119" (UID: "016789a7-277b-4108-8648-407ff0784119"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:04.765273 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.765207    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:08:04.765273 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.765224    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:08:04.766914 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.766891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016789a7-277b-4108-8648-407ff0784119-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "016789a7-277b-4108-8648-407ff0784119" (UID: "016789a7-277b-4108-8648-407ff0784119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:08:04.767044 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.767024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016789a7-277b-4108-8648-407ff0784119-kube-api-access-ts57f" (OuterVolumeSpecName: "kube-api-access-ts57f") pod "016789a7-277b-4108-8648-407ff0784119" (UID: "016789a7-277b-4108-8648-407ff0784119"). InnerVolumeSpecName "kube-api-access-ts57f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:08:04.767044 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.767036    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-dshm" (OuterVolumeSpecName: "dshm") pod "016789a7-277b-4108-8648-407ff0784119" (UID: "016789a7-277b-4108-8648-407ff0784119"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:04.866197 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.866170    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:08:04.866197 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.866197    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts57f\" (UniqueName: \"kubernetes.io/projected/016789a7-277b-4108-8648-407ff0784119-kube-api-access-ts57f\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:08:04.866361 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.866207    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/016789a7-277b-4108-8648-407ff0784119-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:08:04.866361 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:04.866215    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/016789a7-277b-4108-8648-407ff0784119-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:08:05.036401 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:05.036366    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"]
Sep 12 03:08:05.040044 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:05.040018    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/sub-test-model-e401bd-kserve-7fb4d9bbcf-zq42r"]
Sep 12 03:08:05.249398 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:05.249362    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="016789a7-277b-4108-8648-407ff0784119" path="/var/lib/kubelet/pods/016789a7-277b-4108-8648-407ff0784119/volumes"
Sep 12 03:08:06.539418 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:06.539375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:08:13.436344 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:13.436310    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:13.436813 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:13.436355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:13.449073 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:13.449047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:13.770079 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:13.769993    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:08:48.602830 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.602795    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"]
Sep 12 03:08:48.603447 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.603426    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="016789a7-277b-4108-8648-407ff0784119" containerName="main"
Sep 12 03:08:48.603529 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.603451    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="016789a7-277b-4108-8648-407ff0784119" containerName="main"
Sep 12 03:08:48.603585 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.603570    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="016789a7-277b-4108-8648-407ff0784119" containerName="main"
Sep 12 03:08:48.606750 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.606730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.609555 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.609535    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"kube-root-ca.crt\""
Sep 12 03:08:48.610698 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.610679    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"default-dockercfg-f52m4\""
Sep 12 03:08:48.610829 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.610760    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"openshift-service-ca.crt\""
Sep 12 03:08:48.610899 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.610864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e401bd\"/\"rate-test-model-e401bd-kserve-self-signed-certs\""
Sep 12 03:08:48.615709 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.615688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"]
Sep 12 03:08:48.655605 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.655577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcwjt\" (UniqueName: \"kubernetes.io/projected/8c264a9a-8228-4427-9ab5-8dfe40119b23-kube-api-access-rcwjt\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.655761 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.655625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-tmp-dir\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.655761 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.655651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-home\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.655761 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.655741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-dshm\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.655893 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.655767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-model-cache\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.655893 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.655788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c264a9a-8228-4427-9ab5-8dfe40119b23-tls-certs\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.756830 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.756797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-home\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.756984 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.756853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-dshm\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.756984 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.756877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-model-cache\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.756984 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.756905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c264a9a-8228-4427-9ab5-8dfe40119b23-tls-certs\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.756984 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.756966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcwjt\" (UniqueName: \"kubernetes.io/projected/8c264a9a-8228-4427-9ab5-8dfe40119b23-kube-api-access-rcwjt\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.757250 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.757023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-tmp-dir\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.757250 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.757215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-home\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.757362 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.757287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-model-cache\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.757426 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.757404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-tmp-dir\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.759272 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.759253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-dshm\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.759456 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.759440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c264a9a-8228-4427-9ab5-8dfe40119b23-tls-certs\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.764337 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.764318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcwjt\" (UniqueName: \"kubernetes.io/projected/8c264a9a-8228-4427-9ab5-8dfe40119b23-kube-api-access-rcwjt\") pod \"rate-test-model-e401bd-kserve-d7f8597d7-8w89p\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") " pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:48.917619 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:48.917532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:49.194785 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:49.194685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"]
Sep 12 03:08:49.197716 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:08:49.197680    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c264a9a_8228_4427_9ab5_8dfe40119b23.slice/crio-148d7fd20970b826d4416ab692015c21e7becd393ed17a3a08d75d19f05acba2 WatchSource:0}: Error finding container 148d7fd20970b826d4416ab692015c21e7becd393ed17a3a08d75d19f05acba2: Status 404 returned error can't find the container with id 148d7fd20970b826d4416ab692015c21e7becd393ed17a3a08d75d19f05acba2
Sep 12 03:08:49.941740 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:49.941709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p" event={"ID":"8c264a9a-8228-4427-9ab5-8dfe40119b23","Type":"ContainerStarted","Data":"a6249e80648e0e8cdc4cb986a41ce0bf6e54fe77d841806b1e2a2605a0f778fa"}
Sep 12 03:08:49.942181 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:49.941749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p" event={"ID":"8c264a9a-8228-4427-9ab5-8dfe40119b23","Type":"ContainerStarted","Data":"148d7fd20970b826d4416ab692015c21e7becd393ed17a3a08d75d19f05acba2"}
Sep 12 03:08:49.958252 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:49.958196    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p" podStartSLOduration=1.958177914 podStartE2EDuration="1.958177914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:08:49.95700005 +0000 UTC m=+1969.225033718" watchObservedRunningTime="2026-09-12 03:08:49.958177914 +0000 UTC m=+1969.226211565"
Sep 12 03:08:52.849806 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:52.849728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:08:58.917872 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:58.917835    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:58.918394 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:58.917896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:58.930811 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:58.930783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:08:58.995855 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:08:58.995824    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:09:08.043316 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:08.043280    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:09:19.146541 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:19.146501    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:09:26.096921 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:26.096884    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"]
Sep 12 03:09:27.033691 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:27.033660    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"]
Sep 12 03:09:31.749772 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:31.749743    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:09:41.139662 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.139625    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924" podUID="463845f6-f53b-453b-9a76-6d9d21616d82" containerName="main" containerID="cri-o://749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea" gracePeriod=45
Sep 12 03:09:41.382228 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.382205    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:09:41.550487 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-tmp-dir\") pod \"463845f6-f53b-453b-9a76-6d9d21616d82\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") "
Sep 12 03:09:41.550657 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-model-cache\") pod \"463845f6-f53b-453b-9a76-6d9d21616d82\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") "
Sep 12 03:09:41.550657 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44r7t\" (UniqueName: \"kubernetes.io/projected/463845f6-f53b-453b-9a76-6d9d21616d82-kube-api-access-44r7t\") pod \"463845f6-f53b-453b-9a76-6d9d21616d82\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") "
Sep 12 03:09:41.550657 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550617    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463845f6-f53b-453b-9a76-6d9d21616d82-tls-certs\") pod \"463845f6-f53b-453b-9a76-6d9d21616d82\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") "
Sep 12 03:09:41.550657 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-dshm\") pod \"463845f6-f53b-453b-9a76-6d9d21616d82\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") "
Sep 12 03:09:41.550838 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-home\") pod \"463845f6-f53b-453b-9a76-6d9d21616d82\" (UID: \"463845f6-f53b-453b-9a76-6d9d21616d82\") "
Sep 12 03:09:41.550838 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "463845f6-f53b-453b-9a76-6d9d21616d82" (UID: "463845f6-f53b-453b-9a76-6d9d21616d82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:41.550838 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-model-cache" (OuterVolumeSpecName: "model-cache") pod "463845f6-f53b-453b-9a76-6d9d21616d82" (UID: "463845f6-f53b-453b-9a76-6d9d21616d82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:41.551003 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550943    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:41.551003 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550955    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:41.551003 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.550993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-home" (OuterVolumeSpecName: "home") pod "463845f6-f53b-453b-9a76-6d9d21616d82" (UID: "463845f6-f53b-453b-9a76-6d9d21616d82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:41.552863 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.552843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-dshm" (OuterVolumeSpecName: "dshm") pod "463845f6-f53b-453b-9a76-6d9d21616d82" (UID: "463845f6-f53b-453b-9a76-6d9d21616d82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:41.552983 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.552964    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463845f6-f53b-453b-9a76-6d9d21616d82-kube-api-access-44r7t" (OuterVolumeSpecName: "kube-api-access-44r7t") pod "463845f6-f53b-453b-9a76-6d9d21616d82" (UID: "463845f6-f53b-453b-9a76-6d9d21616d82"). InnerVolumeSpecName "kube-api-access-44r7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:09:41.553026 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.552975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463845f6-f53b-453b-9a76-6d9d21616d82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "463845f6-f53b-453b-9a76-6d9d21616d82" (UID: "463845f6-f53b-453b-9a76-6d9d21616d82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:09:41.651873 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.651845    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44r7t\" (UniqueName: \"kubernetes.io/projected/463845f6-f53b-453b-9a76-6d9d21616d82-kube-api-access-44r7t\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:41.651873 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.651872    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463845f6-f53b-453b-9a76-6d9d21616d82-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:41.652043 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.651885    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:41.652043 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:41.651893    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463845f6-f53b-453b-9a76-6d9d21616d82-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:42.053607 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.053555    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p" podUID="8c264a9a-8228-4427-9ab5-8dfe40119b23" containerName="main" containerID="cri-o://a6249e80648e0e8cdc4cb986a41ce0bf6e54fe77d841806b1e2a2605a0f778fa" gracePeriod=45
Sep 12 03:09:42.201450 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.201420    2579 generic.go:356] "Generic (PLEG): container finished" podID="463845f6-f53b-453b-9a76-6d9d21616d82" containerID="749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea" exitCode=0
Sep 12 03:09:42.201905 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.201480    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"
Sep 12 03:09:42.201905 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.201522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924" event={"ID":"463845f6-f53b-453b-9a76-6d9d21616d82","Type":"ContainerDied","Data":"749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea"}
Sep 12 03:09:42.201905 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.201576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924" event={"ID":"463845f6-f53b-453b-9a76-6d9d21616d82","Type":"ContainerDied","Data":"7713d7e5a8b852f4a8d5d76a02c7efb5a5ec2a3d9d000f0d7c5d7336d89cdc52"}
Sep 12 03:09:42.201905 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.201606    2579 scope.go:117] "RemoveContainer" containerID="749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea"
Sep 12 03:09:42.203281 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.203256    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c264a9a-8228-4427-9ab5-8dfe40119b23" containerID="a6249e80648e0e8cdc4cb986a41ce0bf6e54fe77d841806b1e2a2605a0f778fa" exitCode=0
Sep 12 03:09:42.203392 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.203354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p" event={"ID":"8c264a9a-8228-4427-9ab5-8dfe40119b23","Type":"ContainerDied","Data":"a6249e80648e0e8cdc4cb986a41ce0bf6e54fe77d841806b1e2a2605a0f778fa"}
Sep 12 03:09:42.211537 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.211391    2579 scope.go:117] "RemoveContainer" containerID="749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea"
Sep 12 03:09:42.211823 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:09:42.211781    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea\": container with ID starting with 749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea not found: ID does not exist" containerID="749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea"
Sep 12 03:09:42.211933 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.211829    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea"} err="failed to get container status \"749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea\": rpc error: code = NotFound desc = could not find container \"749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea\": container with ID starting with 749735fefa06fd79d9fc8b733c0a958b5fb0c9509689f50e50962b79571c9aea not found: ID does not exist"
Sep 12 03:09:42.224210 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.224186    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"]
Sep 12 03:09:42.230138 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.230115    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/rate-test-model-f91169-kserve-6599d9bc55-hh924"]
Sep 12 03:09:42.244018 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.243974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:09:42.316430 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.316365    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:09:42.459528 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459488    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-home\") pod \"8c264a9a-8228-4427-9ab5-8dfe40119b23\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") "
Sep 12 03:09:42.459722 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459562    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c264a9a-8228-4427-9ab5-8dfe40119b23-tls-certs\") pod \"8c264a9a-8228-4427-9ab5-8dfe40119b23\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") "
Sep 12 03:09:42.459722 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459639    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-tmp-dir\") pod \"8c264a9a-8228-4427-9ab5-8dfe40119b23\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") "
Sep 12 03:09:42.459722 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459668    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-model-cache\") pod \"8c264a9a-8228-4427-9ab5-8dfe40119b23\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") "
Sep 12 03:09:42.459722 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-dshm\") pod \"8c264a9a-8228-4427-9ab5-8dfe40119b23\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") "
Sep 12 03:09:42.459722 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcwjt\" (UniqueName: \"kubernetes.io/projected/8c264a9a-8228-4427-9ab5-8dfe40119b23-kube-api-access-rcwjt\") pod \"8c264a9a-8228-4427-9ab5-8dfe40119b23\" (UID: \"8c264a9a-8228-4427-9ab5-8dfe40119b23\") "
Sep 12 03:09:42.459970 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459720    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-home" (OuterVolumeSpecName: "home") pod "8c264a9a-8228-4427-9ab5-8dfe40119b23" (UID: "8c264a9a-8228-4427-9ab5-8dfe40119b23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:42.459970 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c264a9a-8228-4427-9ab5-8dfe40119b23" (UID: "8c264a9a-8228-4427-9ab5-8dfe40119b23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:42.459970 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.459935    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c264a9a-8228-4427-9ab5-8dfe40119b23" (UID: "8c264a9a-8228-4427-9ab5-8dfe40119b23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:42.460138 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.460047    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:42.460138 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.460065    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:42.460138 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.460076    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:42.461795 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.461775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c264a9a-8228-4427-9ab5-8dfe40119b23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c264a9a-8228-4427-9ab5-8dfe40119b23" (UID: "8c264a9a-8228-4427-9ab5-8dfe40119b23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:09:42.461870 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.461842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c264a9a-8228-4427-9ab5-8dfe40119b23-kube-api-access-rcwjt" (OuterVolumeSpecName: "kube-api-access-rcwjt") pod "8c264a9a-8228-4427-9ab5-8dfe40119b23" (UID: "8c264a9a-8228-4427-9ab5-8dfe40119b23"). InnerVolumeSpecName "kube-api-access-rcwjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:09:42.461935 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.461922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-dshm" (OuterVolumeSpecName: "dshm") pod "8c264a9a-8228-4427-9ab5-8dfe40119b23" (UID: "8c264a9a-8228-4427-9ab5-8dfe40119b23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:09:42.561384 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.561357    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c264a9a-8228-4427-9ab5-8dfe40119b23-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:42.561384 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.561380    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcwjt\" (UniqueName: \"kubernetes.io/projected/8c264a9a-8228-4427-9ab5-8dfe40119b23-kube-api-access-rcwjt\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:42.561384 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:42.561391    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c264a9a-8228-4427-9ab5-8dfe40119b23-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:09:43.210140 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:43.210112    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"
Sep 12 03:09:43.210597 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:43.210145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p" event={"ID":"8c264a9a-8228-4427-9ab5-8dfe40119b23","Type":"ContainerDied","Data":"148d7fd20970b826d4416ab692015c21e7becd393ed17a3a08d75d19f05acba2"}
Sep 12 03:09:43.210597 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:43.210192    2579 scope.go:117] "RemoveContainer" containerID="a6249e80648e0e8cdc4cb986a41ce0bf6e54fe77d841806b1e2a2605a0f778fa"
Sep 12 03:09:43.229574 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:43.229550    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"]
Sep 12 03:09:43.233535 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:43.233515    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e401bd/rate-test-model-e401bd-kserve-d7f8597d7-8w89p"]
Sep 12 03:09:43.248567 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:43.248538    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="463845f6-f53b-453b-9a76-6d9d21616d82" path="/var/lib/kubelet/pods/463845f6-f53b-453b-9a76-6d9d21616d82/volumes"
Sep 12 03:09:43.248912 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:09:43.248899    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c264a9a-8228-4427-9ab5-8dfe40119b23" path="/var/lib/kubelet/pods/8c264a9a-8228-4427-9ab5-8dfe40119b23/volumes"
Sep 12 03:10:02.023570 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.023536    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"]
Sep 12 03:10:02.024464 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.024438    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463845f6-f53b-453b-9a76-6d9d21616d82" containerName="main"
Sep 12 03:10:02.024464 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.024463    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="463845f6-f53b-453b-9a76-6d9d21616d82" containerName="main"
Sep 12 03:10:02.024628 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.024478    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c264a9a-8228-4427-9ab5-8dfe40119b23" containerName="main"
Sep 12 03:10:02.024628 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.024488    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c264a9a-8228-4427-9ab5-8dfe40119b23" containerName="main"
Sep 12 03:10:02.024628 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.024602    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c264a9a-8228-4427-9ab5-8dfe40119b23" containerName="main"
Sep 12 03:10:02.024628 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.024618    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="463845f6-f53b-453b-9a76-6d9d21616d82" containerName="main"
Sep 12 03:10:02.029757 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.029734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.031999 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.031979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-27bf83-serving-cert\""
Sep 12 03:10:02.036974 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.036951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"]
Sep 12 03:10:02.131218 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.131180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.131390 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.131230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-metrics-certs\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.131390 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.131336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587fb33-265a-4be6-9144-47a7dab52e61-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.131509 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.131410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xgg\" (UniqueName: \"kubernetes.io/projected/3587fb33-265a-4be6-9144-47a7dab52e61-kube-api-access-g5xgg\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.232108 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.232066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xgg\" (UniqueName: \"kubernetes.io/projected/3587fb33-265a-4be6-9144-47a7dab52e61-kube-api-access-g5xgg\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.232308 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.232116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.232308 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:10:02.232265    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-serving-cert: secret "maas-api-e2e-ipp-b-27bf83-serving-cert" not found
Sep 12 03:10:02.232308 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:10:02.232310    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls podName:3587fb33-265a-4be6-9144-47a7dab52e61 nodeName:}" failed. No retries permitted until 2026-09-12 03:10:02.73229618 +0000 UTC m=+2042.000329809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls") pod "maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d" (UID: "3587fb33-265a-4be6-9144-47a7dab52e61") : secret "maas-api-e2e-ipp-b-27bf83-serving-cert" not found
Sep 12 03:10:02.232501 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.232304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-metrics-certs\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.232501 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.232423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587fb33-265a-4be6-9144-47a7dab52e61-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.233055 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.233032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587fb33-265a-4be6-9144-47a7dab52e61-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.235005 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.234987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-metrics-certs\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.243361 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.243339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xgg\" (UniqueName: \"kubernetes.io/projected/3587fb33-265a-4be6-9144-47a7dab52e61-kube-api-access-g5xgg\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.736790 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.736753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.739376 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.739351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:02.943395 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:02.943354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:03.215124 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:03.215083    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"]
Sep 12 03:10:03.220124 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:10:03.220069    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3587fb33_265a_4be6_9144_47a7dab52e61.slice/crio-782dc6c71d04214b12df3d9a01e0a2e306b64c467474e4224252fd2ceda2f3ba WatchSource:0}: Error finding container 782dc6c71d04214b12df3d9a01e0a2e306b64c467474e4224252fd2ceda2f3ba: Status 404 returned error can't find the container with id 782dc6c71d04214b12df3d9a01e0a2e306b64c467474e4224252fd2ceda2f3ba
Sep 12 03:10:03.220998 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:03.220981    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 03:10:03.305907 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:03.305827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d" event={"ID":"3587fb33-265a-4be6-9144-47a7dab52e61","Type":"ContainerStarted","Data":"782dc6c71d04214b12df3d9a01e0a2e306b64c467474e4224252fd2ceda2f3ba"}
Sep 12 03:10:04.311690 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:04.311656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d" event={"ID":"3587fb33-265a-4be6-9144-47a7dab52e61","Type":"ContainerStarted","Data":"2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6"}
Sep 12 03:10:04.312105 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:04.311731    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:04.329830 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:04.329769    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d" podStartSLOduration=2.9106578560000003 podStartE2EDuration="3.329748286s" totalImagesPullingTime="419.09043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:10:04.327355703 +0000 UTC m=+2043.595389351" watchObservedRunningTime="2026-09-12 03:10:04.329748286 +0000 UTC m=+2043.597781934"
Sep 12 03:10:08.646771 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:08.646735    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:10:10.321740 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:10.321711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:10:44.518870 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.518773    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"]
Sep 12 03:10:44.522670 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.522647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.525273 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.525245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ad5a1b\"/\"openshift-service-ca.crt\""
Sep 12 03:10:44.526497 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.526466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ad5a1b\"/\"ipp-route-ad5a1b-kserve-self-signed-certs\""
Sep 12 03:10:44.526610 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.526520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ad5a1b\"/\"default-dockercfg-rqppk\""
Sep 12 03:10:44.526610 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.526530    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ad5a1b\"/\"kube-root-ca.crt\""
Sep 12 03:10:44.533977 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.533952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"]
Sep 12 03:10:44.707952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.707922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwmf\" (UniqueName: \"kubernetes.io/projected/442e3b0c-ef11-479d-9fe4-04645cf37f4f-kube-api-access-lgwmf\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.707952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.707960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tmp-dir\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.708219 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.707986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-home\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.708219 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.708062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-model-cache\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.708219 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.708127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tls-certs\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.708219 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.708158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-dshm\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809344 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-model-cache\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809344 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tls-certs\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809582 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-dshm\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809582 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwmf\" (UniqueName: \"kubernetes.io/projected/442e3b0c-ef11-479d-9fe4-04645cf37f4f-kube-api-access-lgwmf\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809582 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tmp-dir\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809582 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-home\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809791 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-model-cache\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809791 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tmp-dir\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.809899 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.809843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-home\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.811868 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.811841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-dshm\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.812278 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.812260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tls-certs\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.817973 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.817948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwmf\" (UniqueName: \"kubernetes.io/projected/442e3b0c-ef11-479d-9fe4-04645cf37f4f-kube-api-access-lgwmf\") pod \"ipp-route-ad5a1b-kserve-54fd56f499-xzctb\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") " pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:44.834834 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:44.834816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:45.106275 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:45.106195    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"]
Sep 12 03:10:45.109299 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:10:45.109269    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod442e3b0c_ef11_479d_9fe4_04645cf37f4f.slice/crio-d5b3bef438ea0fddb5e801cc4f4fbe8550515afbadd6bdfcffa8691c63861307 WatchSource:0}: Error finding container d5b3bef438ea0fddb5e801cc4f4fbe8550515afbadd6bdfcffa8691c63861307: Status 404 returned error can't find the container with id d5b3bef438ea0fddb5e801cc4f4fbe8550515afbadd6bdfcffa8691c63861307
Sep 12 03:10:45.514568 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:45.514528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb" event={"ID":"442e3b0c-ef11-479d-9fe4-04645cf37f4f","Type":"ContainerStarted","Data":"1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744"}
Sep 12 03:10:45.514749 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:45.514578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb" event={"ID":"442e3b0c-ef11-479d-9fe4-04645cf37f4f","Type":"ContainerStarted","Data":"d5b3bef438ea0fddb5e801cc4f4fbe8550515afbadd6bdfcffa8691c63861307"}
Sep 12 03:10:45.534970 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:45.534919    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb" podStartSLOduration=1.5349044790000002 podStartE2EDuration="1.534904479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:10:45.533478754 +0000 UTC m=+2084.801512397" watchObservedRunningTime="2026-09-12 03:10:45.534904479 +0000 UTC m=+2084.802938125"
Sep 12 03:10:48.756424 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:48.756390    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:10:54.835543 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:54.835511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:54.836032 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:54.835567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:54.847720 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:54.847690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:10:55.574773 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:10:55.574736    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:11:05.249213 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:05.249184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:11:18.208420 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.208382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"]
Sep 12 03:11:18.212011 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.211994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.214503 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.214482    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-78f530-serving-cert\""
Sep 12 03:11:18.220150 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.220130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"]
Sep 12 03:11:18.303304 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.303265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.303304 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.303308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thmk\" (UniqueName: \"kubernetes.io/projected/ef4fef70-3883-4df1-9874-fe8d7b7181b3-kube-api-access-7thmk\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.303503 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.303338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4fef70-3883-4df1-9874-fe8d7b7181b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.303503 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.303395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.404484 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.404452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.404737 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.404708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.404847 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.404817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thmk\" (UniqueName: \"kubernetes.io/projected/ef4fef70-3883-4df1-9874-fe8d7b7181b3-kube-api-access-7thmk\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.404930 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.404881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4fef70-3883-4df1-9874-fe8d7b7181b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.405508 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.405489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4fef70-3883-4df1-9874-fe8d7b7181b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.407077 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.407055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.407350 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.407329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.412599 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.412582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thmk\" (UniqueName: \"kubernetes.io/projected/ef4fef70-3883-4df1-9874-fe8d7b7181b3-kube-api-access-7thmk\") pod \"maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.524519 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.524439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:18.798938 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:18.798910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"]
Sep 12 03:11:18.800961 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:11:18.800935    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4fef70_3883_4df1_9874_fe8d7b7181b3.slice/crio-6d6543fe617ef0da06d58a29a599f611c7647cd39002c6f39d5163a573d33ed0 WatchSource:0}: Error finding container 6d6543fe617ef0da06d58a29a599f611c7647cd39002c6f39d5163a573d33ed0: Status 404 returned error can't find the container with id 6d6543fe617ef0da06d58a29a599f611c7647cd39002c6f39d5163a573d33ed0
Sep 12 03:11:19.680360 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:19.680330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls" event={"ID":"ef4fef70-3883-4df1-9874-fe8d7b7181b3","Type":"ContainerStarted","Data":"7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15"}
Sep 12 03:11:19.680774 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:19.680386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:19.680774 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:19.680403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls" event={"ID":"ef4fef70-3883-4df1-9874-fe8d7b7181b3","Type":"ContainerStarted","Data":"6d6543fe617ef0da06d58a29a599f611c7647cd39002c6f39d5163a573d33ed0"}
Sep 12 03:11:19.700054 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:19.700005    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls" podStartSLOduration=1.294341535 podStartE2EDuration="1.699989438s" totalImagesPullingTime="405.647903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:11:19.69830543 +0000 UTC m=+2118.966339170" watchObservedRunningTime="2026-09-12 03:11:19.699989438 +0000 UTC m=+2118.968023085"
Sep 12 03:11:23.194885 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:23.194836    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:11:25.691619 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:25.691586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:56.848915 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:56.848826    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"]
Sep 12 03:11:56.849620 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:56.849174    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls" podUID="ef4fef70-3883-4df1-9874-fe8d7b7181b3" containerName="maas-api" containerID="cri-o://7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15" gracePeriod=30
Sep 12 03:11:57.101138 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.101027    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:57.152382 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.152345    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4fef70-3883-4df1-9874-fe8d7b7181b3-openshift-service-ca\") pod \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") "
Sep 12 03:11:57.152382 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.152403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-maas-api-tls\") pod \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") "
Sep 12 03:11:57.152671 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.152430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7thmk\" (UniqueName: \"kubernetes.io/projected/ef4fef70-3883-4df1-9874-fe8d7b7181b3-kube-api-access-7thmk\") pod \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") "
Sep 12 03:11:57.152671 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.152548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-metrics-certs\") pod \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\" (UID: \"ef4fef70-3883-4df1-9874-fe8d7b7181b3\") "
Sep 12 03:11:57.152790 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.152775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4fef70-3883-4df1-9874-fe8d7b7181b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef4fef70-3883-4df1-9874-fe8d7b7181b3" (UID: "ef4fef70-3883-4df1-9874-fe8d7b7181b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:11:57.153037 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.153017    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef4fef70-3883-4df1-9874-fe8d7b7181b3-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:11:57.154970 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.154920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4fef70-3883-4df1-9874-fe8d7b7181b3-kube-api-access-7thmk" (OuterVolumeSpecName: "kube-api-access-7thmk") pod "ef4fef70-3883-4df1-9874-fe8d7b7181b3" (UID: "ef4fef70-3883-4df1-9874-fe8d7b7181b3"). InnerVolumeSpecName "kube-api-access-7thmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:11:57.155437 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.155403    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef4fef70-3883-4df1-9874-fe8d7b7181b3" (UID: "ef4fef70-3883-4df1-9874-fe8d7b7181b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:11:57.155530 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.155460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef4fef70-3883-4df1-9874-fe8d7b7181b3" (UID: "ef4fef70-3883-4df1-9874-fe8d7b7181b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:11:57.253967 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.253920    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:11:57.253967 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.253955    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef4fef70-3883-4df1-9874-fe8d7b7181b3-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:11:57.253967 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.253968    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thmk\" (UniqueName: \"kubernetes.io/projected/ef4fef70-3883-4df1-9874-fe8d7b7181b3-kube-api-access-7thmk\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:11:57.877795 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.877760    2579 generic.go:356] "Generic (PLEG): container finished" podID="ef4fef70-3883-4df1-9874-fe8d7b7181b3" containerID="7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15" exitCode=0
Sep 12 03:11:57.878308 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.877815    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"
Sep 12 03:11:57.878410 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.877852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls" event={"ID":"ef4fef70-3883-4df1-9874-fe8d7b7181b3","Type":"ContainerDied","Data":"7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15"}
Sep 12 03:11:57.878559 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.878525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls" event={"ID":"ef4fef70-3883-4df1-9874-fe8d7b7181b3","Type":"ContainerDied","Data":"6d6543fe617ef0da06d58a29a599f611c7647cd39002c6f39d5163a573d33ed0"}
Sep 12 03:11:57.878620 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.878609    2579 scope.go:117] "RemoveContainer" containerID="7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15"
Sep 12 03:11:57.896795 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.896760    2579 scope.go:117] "RemoveContainer" containerID="7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15"
Sep 12 03:11:57.897374 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:11:57.897333    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15\": container with ID starting with 7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15 not found: ID does not exist" containerID="7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15"
Sep 12 03:11:57.897456 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.897395    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15"} err="failed to get container status \"7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15\": rpc error: code = NotFound desc = could not find container \"7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15\": container with ID starting with 7a49ea943c608d5c432ae97cf8d816dd59bdb35d7776e77eff7b628c45504a15 not found: ID does not exist"
Sep 12 03:11:57.902159 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.902125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"]
Sep 12 03:11:57.904542 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:57.904515    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78f530-57c5975d67-d8cls"]
Sep 12 03:11:59.250508 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:11:59.250478    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef4fef70-3883-4df1-9874-fe8d7b7181b3" path="/var/lib/kubelet/pods/ef4fef70-3883-4df1-9874-fe8d7b7181b3/volumes"
Sep 12 03:12:01.649185 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:01.649151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:12:21.451353 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:21.451310    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:12:28.717616 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:28.717561    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"]
Sep 12 03:12:35.040993 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:35.040951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:12:43.738373 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:43.738335    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb" podUID="442e3b0c-ef11-479d-9fe4-04645cf37f4f" containerName="main" containerID="cri-o://1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744" gracePeriod=45
Sep 12 03:12:43.997130 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:43.997009    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:12:44.108068 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108023    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tmp-dir\") pod \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") "
Sep 12 03:12:44.108068 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-dshm\") pod \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") "
Sep 12 03:12:44.108347 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-home\") pod \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") "
Sep 12 03:12:44.108347 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-model-cache\") pod \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") "
Sep 12 03:12:44.108347 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108218    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgwmf\" (UniqueName: \"kubernetes.io/projected/442e3b0c-ef11-479d-9fe4-04645cf37f4f-kube-api-access-lgwmf\") pod \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") "
Sep 12 03:12:44.108347 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tls-certs\") pod \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\" (UID: \"442e3b0c-ef11-479d-9fe4-04645cf37f4f\") "
Sep 12 03:12:44.108561 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108403    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "442e3b0c-ef11-479d-9fe4-04645cf37f4f" (UID: "442e3b0c-ef11-479d-9fe4-04645cf37f4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:44.108561 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-home" (OuterVolumeSpecName: "home") pod "442e3b0c-ef11-479d-9fe4-04645cf37f4f" (UID: "442e3b0c-ef11-479d-9fe4-04645cf37f4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:44.108561 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "442e3b0c-ef11-479d-9fe4-04645cf37f4f" (UID: "442e3b0c-ef11-479d-9fe4-04645cf37f4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:44.108728 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108591    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:12:44.108728 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108604    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:12:44.108728 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.108634    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:12:44.110762 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.110724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-dshm" (OuterVolumeSpecName: "dshm") pod "442e3b0c-ef11-479d-9fe4-04645cf37f4f" (UID: "442e3b0c-ef11-479d-9fe4-04645cf37f4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:44.110889 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.110724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "442e3b0c-ef11-479d-9fe4-04645cf37f4f" (UID: "442e3b0c-ef11-479d-9fe4-04645cf37f4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:12:44.110889 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.110779    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442e3b0c-ef11-479d-9fe4-04645cf37f4f-kube-api-access-lgwmf" (OuterVolumeSpecName: "kube-api-access-lgwmf") pod "442e3b0c-ef11-479d-9fe4-04645cf37f4f" (UID: "442e3b0c-ef11-479d-9fe4-04645cf37f4f"). InnerVolumeSpecName "kube-api-access-lgwmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:12:44.126146 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.126112    2579 generic.go:356] "Generic (PLEG): container finished" podID="442e3b0c-ef11-479d-9fe4-04645cf37f4f" containerID="1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744" exitCode=0
Sep 12 03:12:44.126321 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.126160    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"
Sep 12 03:12:44.126321 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.126151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb" event={"ID":"442e3b0c-ef11-479d-9fe4-04645cf37f4f","Type":"ContainerDied","Data":"1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744"}
Sep 12 03:12:44.126321 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.126274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb" event={"ID":"442e3b0c-ef11-479d-9fe4-04645cf37f4f","Type":"ContainerDied","Data":"d5b3bef438ea0fddb5e801cc4f4fbe8550515afbadd6bdfcffa8691c63861307"}
Sep 12 03:12:44.126321 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.126297    2579 scope.go:117] "RemoveContainer" containerID="1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744"
Sep 12 03:12:44.136424 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.136396    2579 scope.go:117] "RemoveContainer" containerID="1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744"
Sep 12 03:12:44.136788 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:12:44.136768    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744\": container with ID starting with 1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744 not found: ID does not exist" containerID="1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744"
Sep 12 03:12:44.136841 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.136801    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744"} err="failed to get container status \"1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744\": rpc error: code = NotFound desc = could not find container \"1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744\": container with ID starting with 1ce884cdd5ffe2d16c1e068d3b5dd487e29c2564891365598d925fd1f41a6744 not found: ID does not exist"
Sep 12 03:12:44.147997 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.147956    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"]
Sep 12 03:12:44.153520 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.153486    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ad5a1b/ipp-route-ad5a1b-kserve-54fd56f499-xzctb"]
Sep 12 03:12:44.209880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.209835    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442e3b0c-ef11-479d-9fe4-04645cf37f4f-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:12:44.209880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.209869    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgwmf\" (UniqueName: \"kubernetes.io/projected/442e3b0c-ef11-479d-9fe4-04645cf37f4f-kube-api-access-lgwmf\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:12:44.209880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:44.209879    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442e3b0c-ef11-479d-9fe4-04645cf37f4f-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:12:45.250063 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:12:45.250024    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442e3b0c-ef11-479d-9fe4-04645cf37f4f" path="/var/lib/kubelet/pods/442e3b0c-ef11-479d-9fe4-04645cf37f4f/volumes"
Sep 12 03:13:22.736015 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:22.735914    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"]
Sep 12 03:13:22.736674 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:22.736261    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d" podUID="3587fb33-265a-4be6-9144-47a7dab52e61" containerName="maas-api" containerID="cri-o://2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6" gracePeriod=30
Sep 12 03:13:22.999661 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:22.999572    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:13:23.076177 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.076142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587fb33-265a-4be6-9144-47a7dab52e61-openshift-service-ca\") pod \"3587fb33-265a-4be6-9144-47a7dab52e61\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") "
Sep 12 03:13:23.076390 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.076211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-metrics-certs\") pod \"3587fb33-265a-4be6-9144-47a7dab52e61\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") "
Sep 12 03:13:23.076390 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.076247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5xgg\" (UniqueName: \"kubernetes.io/projected/3587fb33-265a-4be6-9144-47a7dab52e61-kube-api-access-g5xgg\") pod \"3587fb33-265a-4be6-9144-47a7dab52e61\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") "
Sep 12 03:13:23.076390 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.076355    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls\") pod \"3587fb33-265a-4be6-9144-47a7dab52e61\" (UID: \"3587fb33-265a-4be6-9144-47a7dab52e61\") "
Sep 12 03:13:23.076608 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.076582    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3587fb33-265a-4be6-9144-47a7dab52e61-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3587fb33-265a-4be6-9144-47a7dab52e61" (UID: "3587fb33-265a-4be6-9144-47a7dab52e61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:23.076672 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.076659    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3587fb33-265a-4be6-9144-47a7dab52e61-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:23.078783 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.078739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3587fb33-265a-4be6-9144-47a7dab52e61" (UID: "3587fb33-265a-4be6-9144-47a7dab52e61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:13:23.078783 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.078758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3587fb33-265a-4be6-9144-47a7dab52e61" (UID: "3587fb33-265a-4be6-9144-47a7dab52e61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:13:23.078963 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.078752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3587fb33-265a-4be6-9144-47a7dab52e61-kube-api-access-g5xgg" (OuterVolumeSpecName: "kube-api-access-g5xgg") pod "3587fb33-265a-4be6-9144-47a7dab52e61" (UID: "3587fb33-265a-4be6-9144-47a7dab52e61"). InnerVolumeSpecName "kube-api-access-g5xgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:23.177790 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.177745    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:23.177790 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.177783    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xgg\" (UniqueName: \"kubernetes.io/projected/3587fb33-265a-4be6-9144-47a7dab52e61-kube-api-access-g5xgg\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:23.177790 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.177794    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3587fb33-265a-4be6-9144-47a7dab52e61-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:23.335860 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.335828    2579 generic.go:356] "Generic (PLEG): container finished" podID="3587fb33-265a-4be6-9144-47a7dab52e61" containerID="2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6" exitCode=0
Sep 12 03:13:23.336107 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.335879    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"
Sep 12 03:13:23.336107 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.335879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d" event={"ID":"3587fb33-265a-4be6-9144-47a7dab52e61","Type":"ContainerDied","Data":"2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6"}
Sep 12 03:13:23.336107 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.336016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d" event={"ID":"3587fb33-265a-4be6-9144-47a7dab52e61","Type":"ContainerDied","Data":"782dc6c71d04214b12df3d9a01e0a2e306b64c467474e4224252fd2ceda2f3ba"}
Sep 12 03:13:23.336107 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.336054    2579 scope.go:117] "RemoveContainer" containerID="2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6"
Sep 12 03:13:23.345989 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.345966    2579 scope.go:117] "RemoveContainer" containerID="2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6"
Sep 12 03:13:23.346408 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:13:23.346375    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6\": container with ID starting with 2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6 not found: ID does not exist" containerID="2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6"
Sep 12 03:13:23.346520 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.346418    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6"} err="failed to get container status \"2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6\": rpc error: code = NotFound desc = could not find container \"2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6\": container with ID starting with 2fbfd77c3bec5b9900b38793163ba18cd43002f9e8f0c9a5b9dad4338acdfbc6 not found: ID does not exist"
Sep 12 03:13:23.352496 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.352458    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"]
Sep 12 03:13:23.358488 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:23.358444    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27bf83-5f5b58b94d-7sv8d"]
Sep 12 03:13:25.250086 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:25.250044    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3587fb33-265a-4be6-9144-47a7dab52e61" path="/var/lib/kubelet/pods/3587fb33-265a-4be6-9144-47a7dab52e61/volumes"
Sep 12 03:13:26.341013 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:26.340963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:13:40.223739 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.223695    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"]
Sep 12 03:13:40.224450 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.223961    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr" podUID="c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" containerName="maas-api" containerID="cri-o://530fb1877cae96018cbbb1843ee227979de874825dea4a4fbe581acea2a4328f" gracePeriod=30
Sep 12 03:13:40.434360 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.434315    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" containerID="530fb1877cae96018cbbb1843ee227979de874825dea4a4fbe581acea2a4328f" exitCode=0
Sep 12 03:13:40.434581 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.434414    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr" event={"ID":"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4","Type":"ContainerDied","Data":"530fb1877cae96018cbbb1843ee227979de874825dea4a4fbe581acea2a4328f"}
Sep 12 03:13:40.481828 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.481742    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:13:40.543369 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.543331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls\") pod \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") "
Sep 12 03:13:40.543369 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.543373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-metrics-certs\") pod \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") "
Sep 12 03:13:40.543601 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.543410    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-openshift-service-ca\") pod \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") "
Sep 12 03:13:40.543601 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.543467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9tv6\" (UniqueName: \"kubernetes.io/projected/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-kube-api-access-h9tv6\") pod \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\" (UID: \"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4\") "
Sep 12 03:13:40.543811 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.543786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" (UID: "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:40.545894 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.545859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" (UID: "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:13:40.546030 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.545886    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" (UID: "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:13:40.546030 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.545860    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-kube-api-access-h9tv6" (OuterVolumeSpecName: "kube-api-access-h9tv6") pod "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" (UID: "c7d44a7f-8e02-4fff-89d6-0204b10fb1c4"). InnerVolumeSpecName "kube-api-access-h9tv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:40.644535 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.644493    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9tv6\" (UniqueName: \"kubernetes.io/projected/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-kube-api-access-h9tv6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:40.644535 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.644531    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:40.644535 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.644540    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:40.644535 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:40.644549    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:41.441075 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:41.441043    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"
Sep 12 03:13:41.441612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:41.441040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr" event={"ID":"c7d44a7f-8e02-4fff-89d6-0204b10fb1c4","Type":"ContainerDied","Data":"03c3ba537b5091761c21603670e40d4c2772d9715eda2db71fbb71a163e1682a"}
Sep 12 03:13:41.441612 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:41.441198    2579 scope.go:117] "RemoveContainer" containerID="530fb1877cae96018cbbb1843ee227979de874825dea4a4fbe581acea2a4328f"
Sep 12 03:13:41.458685 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:41.458645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"]
Sep 12 03:13:41.463017 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:41.462980    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f91169-85b4b9d884-4chkr"]
Sep 12 03:13:43.249644 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:43.249604    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" path="/var/lib/kubelet/pods/c7d44a7f-8e02-4fff-89d6-0204b10fb1c4/volumes"
Sep 12 03:13:43.441953 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:43.441910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:13:57.893561 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:57.893515    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"]
Sep 12 03:13:57.894415 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:57.894345    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb" podUID="1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" containerName="maas-api" containerID="cri-o://5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807" gracePeriod=30
Sep 12 03:13:58.156972 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.156877    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:13:58.211222 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.211172    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68kkc\" (UniqueName: \"kubernetes.io/projected/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-kube-api-access-68kkc\") pod \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") "
Sep 12 03:13:58.211442 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.211244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-openshift-service-ca\") pod \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") "
Sep 12 03:13:58.211442 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.211292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-maas-api-tls\") pod \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") "
Sep 12 03:13:58.211578 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.211458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-metrics-certs\") pod \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\" (UID: \"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e\") "
Sep 12 03:13:58.211733 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.211691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" (UID: "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:58.211852 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.211834    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:58.213806 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.213772    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" (UID: "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:13:58.214194 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.214170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" (UID: "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:13:58.214276 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.214212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-kube-api-access-68kkc" (OuterVolumeSpecName: "kube-api-access-68kkc") pod "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" (UID: "1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e"). InnerVolumeSpecName "kube-api-access-68kkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:58.312943 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.312903    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:58.312943 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.312935    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68kkc\" (UniqueName: \"kubernetes.io/projected/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-kube-api-access-68kkc\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:58.312943 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.312944    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:13:58.542957 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.542915    2579 generic.go:356] "Generic (PLEG): container finished" podID="1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" containerID="5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807" exitCode=0
Sep 12 03:13:58.543176 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.542974    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"
Sep 12 03:13:58.543176 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.543004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb" event={"ID":"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e","Type":"ContainerDied","Data":"5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807"}
Sep 12 03:13:58.543176 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.543069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb" event={"ID":"1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e","Type":"ContainerDied","Data":"61cfb1be88eaeba77f8a0a025ca58aa9a7f3cc360e4dc467196a5d38a05b41f3"}
Sep 12 03:13:58.543176 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.543134    2579 scope.go:117] "RemoveContainer" containerID="5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807"
Sep 12 03:13:58.553559 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.553534    2579 scope.go:117] "RemoveContainer" containerID="5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807"
Sep 12 03:13:58.553926 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:13:58.553907    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807\": container with ID starting with 5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807 not found: ID does not exist" containerID="5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807"
Sep 12 03:13:58.553996 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.553935    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807"} err="failed to get container status \"5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807\": rpc error: code = NotFound desc = could not find container \"5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807\": container with ID starting with 5aa59efdb8c50b021e55cadf7f70524fa573b221d7dfcdf579e0af0aff719807 not found: ID does not exist"
Sep 12 03:13:58.566160 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.566120    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"]
Sep 12 03:13:58.569513 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:58.569474    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e401bd-c5fd94c5c-6xpvb"]
Sep 12 03:13:59.249811 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:13:59.249770    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" path="/var/lib/kubelet/pods/1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e/volumes"
Sep 12 03:14:00.842995 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:14:00.842953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:14:11.050944 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:14:11.050895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:15:00.136503 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm"]
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136830    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442e3b0c-ef11-479d-9fe4-04645cf37f4f" containerName="main"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136843    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="442e3b0c-ef11-479d-9fe4-04645cf37f4f" containerName="main"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136857    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3587fb33-265a-4be6-9144-47a7dab52e61" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136863    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3587fb33-265a-4be6-9144-47a7dab52e61" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136878    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136884    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136891    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136896    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136908    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef4fef70-3883-4df1-9874-fe8d7b7181b3" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136912    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef4fef70-3883-4df1-9874-fe8d7b7181b3" containerName="maas-api"
Sep 12 03:15:00.136964 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136967    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1c44b5-6d09-4cf0-ab3f-e83b5975f54e" containerName="maas-api"
Sep 12 03:15:00.137355 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3587fb33-265a-4be6-9144-47a7dab52e61" containerName="maas-api"
Sep 12 03:15:00.137355 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136983    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="442e3b0c-ef11-479d-9fe4-04645cf37f4f" containerName="main"
Sep 12 03:15:00.137355 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136991    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d44a7f-8e02-4fff-89d6-0204b10fb1c4" containerName="maas-api"
Sep 12 03:15:00.137355 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.136997    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef4fef70-3883-4df1-9874-fe8d7b7181b3" containerName="maas-api"
Sep 12 03:15:00.140356 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.140332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm"
Sep 12 03:15:00.143148 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.143079    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4g6hp\""
Sep 12 03:15:00.146908 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.146872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm"]
Sep 12 03:15:00.153242 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.153216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm"
Sep 12 03:15:00.428404 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.428372    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm"]
Sep 12 03:15:00.431084 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:15:00.431047    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bf21c7_dbae_46ec_a227_c4b90970f43a.slice/crio-bb908826e034159e762f4dfe9d7ecea1f8e06d1e8b1d59f52d465e23c926628d WatchSource:0}: Error finding container bb908826e034159e762f4dfe9d7ecea1f8e06d1e8b1d59f52d465e23c926628d: Status 404 returned error can't find the container with id bb908826e034159e762f4dfe9d7ecea1f8e06d1e8b1d59f52d465e23c926628d
Sep 12 03:15:00.872649 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.872611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm" event={"ID":"f7bf21c7-dbae-46ec-a227-c4b90970f43a","Type":"ContainerStarted","Data":"5c9155f485e933308af39fdaa0300207c4449640c07f593e552302fe63139867"}
Sep 12 03:15:00.872844 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.872664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm" event={"ID":"f7bf21c7-dbae-46ec-a227-c4b90970f43a","Type":"ContainerStarted","Data":"bb908826e034159e762f4dfe9d7ecea1f8e06d1e8b1d59f52d465e23c926628d"}
Sep 12 03:15:00.888487 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:00.888415    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm" podStartSLOduration=0.888388777 podStartE2EDuration="888.388777ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:15:00.886637847 +0000 UTC m=+2340.154671495" watchObservedRunningTime="2026-09-12 03:15:00.888388777 +0000 UTC m=+2340.156422425"
Sep 12 03:15:01.879106 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:01.879053    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7bf21c7-dbae-46ec-a227-c4b90970f43a" containerID="5c9155f485e933308af39fdaa0300207c4449640c07f593e552302fe63139867" exitCode=0
Sep 12 03:15:01.879525 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:01.879130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm" event={"ID":"f7bf21c7-dbae-46ec-a227-c4b90970f43a","Type":"ContainerDied","Data":"5c9155f485e933308af39fdaa0300207c4449640c07f593e552302fe63139867"}
Sep 12 03:15:03.019580 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:03.019535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm"
Sep 12 03:15:03.891337 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:03.891308    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm"
Sep 12 03:15:03.891534 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:03.891303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819715-7g4nm" event={"ID":"f7bf21c7-dbae-46ec-a227-c4b90970f43a","Type":"ContainerDied","Data":"bb908826e034159e762f4dfe9d7ecea1f8e06d1e8b1d59f52d465e23c926628d"}
Sep 12 03:15:03.891534 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:03.891431    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb908826e034159e762f4dfe9d7ecea1f8e06d1e8b1d59f52d465e23c926628d"
Sep 12 03:15:30.540607 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:30.540563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:15:56.440042 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:15:56.439995    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:16:05.042434 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:05.042377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:16:14.253892 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:14.253851    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:16:29.443921 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:29.443835    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:16:37.935207 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:37.935161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:16:45.376895 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.376839    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"]
Sep 12 03:16:45.377559 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.377199    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9" podUID="aab37b6c-9946-4cc1-a864-de46312b11e3" containerName="manager" containerID="cri-o://36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a" gracePeriod=10
Sep 12 03:16:45.637863 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.637780    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 03:16:45.655446 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.655407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab37b6c-9946-4cc1-a864-de46312b11e3-extensions-socket-volume\") pod \"aab37b6c-9946-4cc1-a864-de46312b11e3\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") "
Sep 12 03:16:45.655634 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.655532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wtc6\" (UniqueName: \"kubernetes.io/projected/aab37b6c-9946-4cc1-a864-de46312b11e3-kube-api-access-5wtc6\") pod \"aab37b6c-9946-4cc1-a864-de46312b11e3\" (UID: \"aab37b6c-9946-4cc1-a864-de46312b11e3\") "
Sep 12 03:16:45.655895 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.655863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab37b6c-9946-4cc1-a864-de46312b11e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aab37b6c-9946-4cc1-a864-de46312b11e3" (UID: "aab37b6c-9946-4cc1-a864-de46312b11e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:16:45.657828 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.657803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab37b6c-9946-4cc1-a864-de46312b11e3-kube-api-access-5wtc6" (OuterVolumeSpecName: "kube-api-access-5wtc6") pod "aab37b6c-9946-4cc1-a864-de46312b11e3" (UID: "aab37b6c-9946-4cc1-a864-de46312b11e3"). InnerVolumeSpecName "kube-api-access-5wtc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:16:45.756584 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.756543    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wtc6\" (UniqueName: \"kubernetes.io/projected/aab37b6c-9946-4cc1-a864-de46312b11e3-kube-api-access-5wtc6\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:16:45.756584 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:45.756580    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab37b6c-9946-4cc1-a864-de46312b11e3-extensions-socket-volume\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:16:46.429193 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.429154    2579 generic.go:356] "Generic (PLEG): container finished" podID="aab37b6c-9946-4cc1-a864-de46312b11e3" containerID="36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a" exitCode=0
Sep 12 03:16:46.429656 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.429200    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"
Sep 12 03:16:46.429656 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.429237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9" event={"ID":"aab37b6c-9946-4cc1-a864-de46312b11e3","Type":"ContainerDied","Data":"36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a"}
Sep 12 03:16:46.429656 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.429289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9" event={"ID":"aab37b6c-9946-4cc1-a864-de46312b11e3","Type":"ContainerDied","Data":"863f1a3f442bc44f384ba182d915b55f92b4471bfd1accd844cac1aeb70b4e6e"}
Sep 12 03:16:46.429656 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.429311    2579 scope.go:117] "RemoveContainer" containerID="36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a"
Sep 12 03:16:46.439528 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.439508    2579 scope.go:117] "RemoveContainer" containerID="36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a"
Sep 12 03:16:46.439868 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:16:46.439848    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a\": container with ID starting with 36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a not found: ID does not exist" containerID="36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a"
Sep 12 03:16:46.439939 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.439879    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a"} err="failed to get container status \"36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a\": rpc error: code = NotFound desc = could not find container \"36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a\": container with ID starting with 36cf544512e2ebc4222f1b96954e61fca344241550f8342cbe2b31d105c3fa4a not found: ID does not exist"
Sep 12 03:16:46.452363 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.452299    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"]
Sep 12 03:16:46.455961 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:46.455929    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frlq9"]
Sep 12 03:16:47.249951 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:16:47.249905    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aab37b6c-9946-4cc1-a864-de46312b11e3" path="/var/lib/kubelet/pods/aab37b6c-9946-4cc1-a864-de46312b11e3/volumes"
Sep 12 03:17:46.077003 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.076963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"]
Sep 12 03:17:46.077550 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.077492    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab37b6c-9946-4cc1-a864-de46312b11e3" containerName="manager"
Sep 12 03:17:46.077550 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.077510    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab37b6c-9946-4cc1-a864-de46312b11e3" containerName="manager"
Sep 12 03:17:46.077550 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.077522    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bf21c7-dbae-46ec-a227-c4b90970f43a" containerName="cleanup"
Sep 12 03:17:46.077550 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.077529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bf21c7-dbae-46ec-a227-c4b90970f43a" containerName="cleanup"
Sep 12 03:17:46.077701 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.077591    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aab37b6c-9946-4cc1-a864-de46312b11e3" containerName="manager"
Sep 12 03:17:46.077701 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.077600    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bf21c7-dbae-46ec-a227-c4b90970f43a" containerName="cleanup"
Sep 12 03:17:46.080677 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.080656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.083486 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.083459    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjdc8\""
Sep 12 03:17:46.092897 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.092868    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"]
Sep 12 03:17:46.127946 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.127900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00cad22-6379-4e30-843f-ce726ab8f488-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p8f6s\" (UID: \"f00cad22-6379-4e30-843f-ce726ab8f488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.127946 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.127949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtct9\" (UniqueName: \"kubernetes.io/projected/f00cad22-6379-4e30-843f-ce726ab8f488-kube-api-access-vtct9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p8f6s\" (UID: \"f00cad22-6379-4e30-843f-ce726ab8f488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.228519 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.228478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00cad22-6379-4e30-843f-ce726ab8f488-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p8f6s\" (UID: \"f00cad22-6379-4e30-843f-ce726ab8f488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.228519 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.228524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtct9\" (UniqueName: \"kubernetes.io/projected/f00cad22-6379-4e30-843f-ce726ab8f488-kube-api-access-vtct9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p8f6s\" (UID: \"f00cad22-6379-4e30-843f-ce726ab8f488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.228914 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.228890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00cad22-6379-4e30-843f-ce726ab8f488-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p8f6s\" (UID: \"f00cad22-6379-4e30-843f-ce726ab8f488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.237624 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.237595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtct9\" (UniqueName: \"kubernetes.io/projected/f00cad22-6379-4e30-843f-ce726ab8f488-kube-api-access-vtct9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p8f6s\" (UID: \"f00cad22-6379-4e30-843f-ce726ab8f488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.391956 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.391847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:46.672368 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.672340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"]
Sep 12 03:17:46.675726 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:17:46.675686    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00cad22_6379_4e30_843f_ce726ab8f488.slice/crio-4adac0852d5462462cce611a488d7976759975538792731fff176128cb024f65 WatchSource:0}: Error finding container 4adac0852d5462462cce611a488d7976759975538792731fff176128cb024f65: Status 404 returned error can't find the container with id 4adac0852d5462462cce611a488d7976759975538792731fff176128cb024f65
Sep 12 03:17:46.741892 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:46.741861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s" event={"ID":"f00cad22-6379-4e30-843f-ce726ab8f488","Type":"ContainerStarted","Data":"4adac0852d5462462cce611a488d7976759975538792731fff176128cb024f65"}
Sep 12 03:17:47.748459 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:47.748419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s" event={"ID":"f00cad22-6379-4e30-843f-ce726ab8f488","Type":"ContainerStarted","Data":"87f997a255294dbf320160c3f8a0e1a916724d1e07b703d83d3e14bb04e5ff06"}
Sep 12 03:17:47.748909 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:47.748579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:17:47.769931 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:47.769877    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s" podStartSLOduration=1.769859915 podStartE2EDuration="1.769859915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:17:47.766970223 +0000 UTC m=+2507.035003905" watchObservedRunningTime="2026-09-12 03:17:47.769859915 +0000 UTC m=+2507.037893628"
Sep 12 03:17:58.756733 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:17:58.756701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p8f6s"
Sep 12 03:18:09.937946 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:09.937903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:18:16.842821 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:16.842775    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:18:27.036564 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:27.036518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:18:37.643232 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:37.643188    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:18:57.217027 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.216981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"]
Sep 12 03:18:57.221549 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.221519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.224123 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.224069    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b07066f0-kserve-self-signed-certs\""
Sep 12 03:18:57.229418 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.229389    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"]
Sep 12 03:18:57.397582 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.397533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39cd4623-537b-4313-9550-86f2f083dca0-tls-certs\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.397796 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.397595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-home\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.397796 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.397620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-tmp-dir\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.397796 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.397643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8c2\" (UniqueName: \"kubernetes.io/projected/39cd4623-537b-4313-9550-86f2f083dca0-kube-api-access-zm8c2\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.397796 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.397748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-dshm\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.397947 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.397829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-model-cache\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.498694 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.498582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-model-cache\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.498694 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.498642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39cd4623-537b-4313-9550-86f2f083dca0-tls-certs\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.498694 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.498681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-home\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.499015 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.498704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-tmp-dir\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.499015 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.498728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8c2\" (UniqueName: \"kubernetes.io/projected/39cd4623-537b-4313-9550-86f2f083dca0-kube-api-access-zm8c2\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.499015 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.498792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-dshm\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.499229 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.499123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-model-cache\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.499229 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.499161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-home\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.499326 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.499239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-tmp-dir\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.501406 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.501377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-dshm\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.501731 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.501709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39cd4623-537b-4313-9550-86f2f083dca0-tls-certs\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.509700 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.509669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8c2\" (UniqueName: \"kubernetes.io/projected/39cd4623-537b-4313-9550-86f2f083dca0-kube-api-access-zm8c2\") pod \"e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") " pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.535412 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.535366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:18:57.819194 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:57.819167    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"]
Sep 12 03:18:57.821943 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:18:57.821903    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39cd4623_537b_4313_9550_86f2f083dca0.slice/crio-26cf502db953e471656df4774a5bb6228685bf05244d3a59581eb58423aa8231 WatchSource:0}: Error finding container 26cf502db953e471656df4774a5bb6228685bf05244d3a59581eb58423aa8231: Status 404 returned error can't find the container with id 26cf502db953e471656df4774a5bb6228685bf05244d3a59581eb58423aa8231
Sep 12 03:18:58.117698 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:58.117596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw" event={"ID":"39cd4623-537b-4313-9550-86f2f083dca0","Type":"ContainerStarted","Data":"c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be"}
Sep 12 03:18:58.117698 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:58.117651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw" event={"ID":"39cd4623-537b-4313-9550-86f2f083dca0","Type":"ContainerStarted","Data":"26cf502db953e471656df4774a5bb6228685bf05244d3a59581eb58423aa8231"}
Sep 12 03:18:58.134142 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:18:58.134062    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw" podStartSLOduration=1.1340413 podStartE2EDuration="1.1340413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:18:58.132118353 +0000 UTC m=+2577.400152001" watchObservedRunningTime="2026-09-12 03:18:58.1340413 +0000 UTC m=+2577.402074947"
Sep 12 03:19:00.051472 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:00.051427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:19:07.536321 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:07.536275    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:19:07.536321 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:07.536335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:19:07.549339 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:07.549302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:19:08.187592 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:08.187557    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:19:32.742359 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:32.742266    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"]
Sep 12 03:19:34.645763 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:34.645730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:19:43.142856 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:43.142802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:19:47.763976 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:47.763917    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw" podUID="39cd4623-537b-4313-9550-86f2f083dca0" containerName="main" containerID="cri-o://c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be" gracePeriod=45
Sep 12 03:19:48.028052 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.027954    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:19:48.079424 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm8c2\" (UniqueName: \"kubernetes.io/projected/39cd4623-537b-4313-9550-86f2f083dca0-kube-api-access-zm8c2\") pod \"39cd4623-537b-4313-9550-86f2f083dca0\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") "
Sep 12 03:19:48.079424 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-tmp-dir\") pod \"39cd4623-537b-4313-9550-86f2f083dca0\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") "
Sep 12 03:19:48.079712 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079463    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-dshm\") pod \"39cd4623-537b-4313-9550-86f2f083dca0\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") "
Sep 12 03:19:48.079712 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39cd4623-537b-4313-9550-86f2f083dca0-tls-certs\") pod \"39cd4623-537b-4313-9550-86f2f083dca0\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") "
Sep 12 03:19:48.079712 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-model-cache\") pod \"39cd4623-537b-4313-9550-86f2f083dca0\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") "
Sep 12 03:19:48.079712 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-home\") pod \"39cd4623-537b-4313-9550-86f2f083dca0\" (UID: \"39cd4623-537b-4313-9550-86f2f083dca0\") "
Sep 12 03:19:48.079929 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39cd4623-537b-4313-9550-86f2f083dca0" (UID: "39cd4623-537b-4313-9550-86f2f083dca0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:48.079929 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-model-cache" (OuterVolumeSpecName: "model-cache") pod "39cd4623-537b-4313-9550-86f2f083dca0" (UID: "39cd4623-537b-4313-9550-86f2f083dca0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:48.080007 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.079908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-home" (OuterVolumeSpecName: "home") pod "39cd4623-537b-4313-9550-86f2f083dca0" (UID: "39cd4623-537b-4313-9550-86f2f083dca0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:48.080046 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.080024    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-home\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:19:48.080046 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.080039    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-tmp-dir\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:19:48.080133 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.080048    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-model-cache\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:19:48.082123 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.082072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-dshm" (OuterVolumeSpecName: "dshm") pod "39cd4623-537b-4313-9550-86f2f083dca0" (UID: "39cd4623-537b-4313-9550-86f2f083dca0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:48.082123 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.082069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39cd4623-537b-4313-9550-86f2f083dca0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39cd4623-537b-4313-9550-86f2f083dca0" (UID: "39cd4623-537b-4313-9550-86f2f083dca0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:19:48.082315 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.082153    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cd4623-537b-4313-9550-86f2f083dca0-kube-api-access-zm8c2" (OuterVolumeSpecName: "kube-api-access-zm8c2") pod "39cd4623-537b-4313-9550-86f2f083dca0" (UID: "39cd4623-537b-4313-9550-86f2f083dca0"). InnerVolumeSpecName "kube-api-access-zm8c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:19:48.181027 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.180982    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm8c2\" (UniqueName: \"kubernetes.io/projected/39cd4623-537b-4313-9550-86f2f083dca0-kube-api-access-zm8c2\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:19:48.181027 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.181018    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39cd4623-537b-4313-9550-86f2f083dca0-dshm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:19:48.181027 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.181027    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39cd4623-537b-4313-9550-86f2f083dca0-tls-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:19:48.384203 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.384074    2579 generic.go:356] "Generic (PLEG): container finished" podID="39cd4623-537b-4313-9550-86f2f083dca0" containerID="c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be" exitCode=0
Sep 12 03:19:48.384203 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.384143    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"
Sep 12 03:19:48.384203 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.384162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw" event={"ID":"39cd4623-537b-4313-9550-86f2f083dca0","Type":"ContainerDied","Data":"c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be"}
Sep 12 03:19:48.384460 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.384219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw" event={"ID":"39cd4623-537b-4313-9550-86f2f083dca0","Type":"ContainerDied","Data":"26cf502db953e471656df4774a5bb6228685bf05244d3a59581eb58423aa8231"}
Sep 12 03:19:48.384460 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.384238    2579 scope.go:117] "RemoveContainer" containerID="c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be"
Sep 12 03:19:48.394452 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.394408    2579 scope.go:117] "RemoveContainer" containerID="c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be"
Sep 12 03:19:48.394913 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:19:48.394891    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be\": container with ID starting with c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be not found: ID does not exist" containerID="c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be"
Sep 12 03:19:48.394999 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.394924    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be"} err="failed to get container status \"c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be\": rpc error: code = NotFound desc = could not find container \"c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be\": container with ID starting with c063a13a576efce3891c705caa3d83bf9730cfd78b5e2431a80fd85d052997be not found: ID does not exist"
Sep 12 03:19:48.407970 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.407933    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"]
Sep 12 03:19:48.409607 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:48.409583    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b07066f0-kserve-59bc9b8ff-b8bqw"]
Sep 12 03:19:49.250144 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.250084    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39cd4623-537b-4313-9550-86f2f083dca0" path="/var/lib/kubelet/pods/39cd4623-537b-4313-9550-86f2f083dca0/volumes"
Sep 12 03:19:49.943662 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.943615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:19:49.947379 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.947341    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"]
Sep 12 03:19:49.947762 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.947741    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39cd4623-537b-4313-9550-86f2f083dca0" containerName="main"
Sep 12 03:19:49.947762 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.947757    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39cd4623-537b-4313-9550-86f2f083dca0" containerName="main"
Sep 12 03:19:49.947958 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.947842    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39cd4623-537b-4313-9550-86f2f083dca0" containerName="main"
Sep 12 03:19:49.952600 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.952561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:49.956084 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.956051    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 03:19:49.956296 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.956084    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 03:19:49.956296 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.956147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsxrv\""
Sep 12 03:19:49.956296 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.956084    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 03:19:49.956417 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.956402    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 12 03:19:49.960215 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.960154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"]
Sep 12 03:19:49.999175 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.999122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn8vm\" (UniqueName: \"kubernetes.io/projected/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-kube-api-access-fn8vm\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:49.999379 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.999193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-maas-api-tls\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:49.999379 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.999219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-openshift-service-ca\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:49.999379 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:49.999246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-metrics-certs\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.100407 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.100362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-maas-api-tls\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.100407 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.100402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-openshift-service-ca\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.100665 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.100434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-metrics-certs\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.100665 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.100511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn8vm\" (UniqueName: \"kubernetes.io/projected/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-kube-api-access-fn8vm\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.101214 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.101191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-openshift-service-ca\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.103256 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.103229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-metrics-certs\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.103359 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.103295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-maas-api-tls\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.108302 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.108272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn8vm\" (UniqueName: \"kubernetes.io/projected/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-kube-api-access-fn8vm\") pod \"maas-api-bb8f5d89c-svvhj\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.266166 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.266030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:50.546713 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.546682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"]
Sep 12 03:19:50.549523 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:19:50.549482    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9982a2_9bcd_407e_a18a_c77395f9bbf1.slice/crio-5f48f86c417d5883a06098babad5b6120135fd887b64042bd4297f5449c24499 WatchSource:0}: Error finding container 5f48f86c417d5883a06098babad5b6120135fd887b64042bd4297f5449c24499: Status 404 returned error can't find the container with id 5f48f86c417d5883a06098babad5b6120135fd887b64042bd4297f5449c24499
Sep 12 03:19:50.551173 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:50.551148    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 03:19:51.404563 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:51.404517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj" event={"ID":"cd9982a2-9bcd-407e-a18a-c77395f9bbf1","Type":"ContainerStarted","Data":"d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0"}
Sep 12 03:19:51.404976 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:51.404595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:51.404976 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:51.404616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj" event={"ID":"cd9982a2-9bcd-407e-a18a-c77395f9bbf1","Type":"ContainerStarted","Data":"5f48f86c417d5883a06098babad5b6120135fd887b64042bd4297f5449c24499"}
Sep 12 03:19:51.458718 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:51.458660    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj" podStartSLOduration=2.084977057 podStartE2EDuration="2.458637938s" totalImagesPullingTime="373.660881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:19:51.457830065 +0000 UTC m=+2630.725863713" watchObservedRunningTime="2026-09-12 03:19:51.458637938 +0000 UTC m=+2630.726671584"
Sep 12 03:19:57.415228 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.415189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:19:57.653934 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.653889    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"]
Sep 12 03:19:57.658043 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.658011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.668528 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.668494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"]
Sep 12 03:19:57.744825 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.744784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:19:57.774727 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.774688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91953934-f3e8-4e89-8dcd-18cd0b989c24-openshift-service-ca\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.774879 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.774730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91953934-f3e8-4e89-8dcd-18cd0b989c24-maas-api-tls\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.774879 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.774804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4kn\" (UniqueName: \"kubernetes.io/projected/91953934-f3e8-4e89-8dcd-18cd0b989c24-kube-api-access-wf4kn\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.774879 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.774841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91953934-f3e8-4e89-8dcd-18cd0b989c24-metrics-certs\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.875966 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.875919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4kn\" (UniqueName: \"kubernetes.io/projected/91953934-f3e8-4e89-8dcd-18cd0b989c24-kube-api-access-wf4kn\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.876215 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.875989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91953934-f3e8-4e89-8dcd-18cd0b989c24-metrics-certs\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.876215 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.876052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91953934-f3e8-4e89-8dcd-18cd0b989c24-openshift-service-ca\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.876215 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.876070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91953934-f3e8-4e89-8dcd-18cd0b989c24-maas-api-tls\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.876907 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.876877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91953934-f3e8-4e89-8dcd-18cd0b989c24-openshift-service-ca\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.878952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.878928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91953934-f3e8-4e89-8dcd-18cd0b989c24-maas-api-tls\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.879157 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.879139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91953934-f3e8-4e89-8dcd-18cd0b989c24-metrics-certs\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.884111 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.884065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4kn\" (UniqueName: \"kubernetes.io/projected/91953934-f3e8-4e89-8dcd-18cd0b989c24-kube-api-access-wf4kn\") pod \"maas-api-75df99c77c-hqdzj\" (UID: \"91953934-f3e8-4e89-8dcd-18cd0b989c24\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:57.970857 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:57.970818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:58.251300 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:58.251184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"]
Sep 12 03:19:58.255671 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:19:58.255623    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91953934_f3e8_4e89_8dcd_18cd0b989c24.slice/crio-0ef0a7e7863c5d3f48b4bd20bd3a52c7aa25515cc47a532a2a883f98c00e97d8 WatchSource:0}: Error finding container 0ef0a7e7863c5d3f48b4bd20bd3a52c7aa25515cc47a532a2a883f98c00e97d8: Status 404 returned error can't find the container with id 0ef0a7e7863c5d3f48b4bd20bd3a52c7aa25515cc47a532a2a883f98c00e97d8
Sep 12 03:19:58.442391 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:58.442355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj" event={"ID":"91953934-f3e8-4e89-8dcd-18cd0b989c24","Type":"ContainerStarted","Data":"0ef0a7e7863c5d3f48b4bd20bd3a52c7aa25515cc47a532a2a883f98c00e97d8"}
Sep 12 03:19:59.449150 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:59.449111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj" event={"ID":"91953934-f3e8-4e89-8dcd-18cd0b989c24","Type":"ContainerStarted","Data":"c3664b3ae7838e9ede2d6c5c403ce0c55314a0ca4f137a2941ff7b844384afbe"}
Sep 12 03:19:59.449614 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:59.449182    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:19:59.466347 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:19:59.466283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj" podStartSLOduration=2.168608091 podStartE2EDuration="2.466260844s" totalImagesPullingTime="297.652753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:19:59.464822732 +0000 UTC m=+2638.732856380" watchObservedRunningTime="2026-09-12 03:19:59.466260844 +0000 UTC m=+2638.734294490"
Sep 12 03:20:04.846889 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:04.846843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:20:05.460528 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:05.460495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75df99c77c-hqdzj"
Sep 12 03:20:05.506040 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:05.506006    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"]
Sep 12 03:20:05.506293 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:05.506240    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj" podUID="cd9982a2-9bcd-407e-a18a-c77395f9bbf1" containerName="maas-api" containerID="cri-o://d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0" gracePeriod=30
Sep 12 03:20:06.755086 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.755057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:20:06.868707 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.868558    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-maas-api-tls\") pod \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") "
Sep 12 03:20:06.868707 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.868689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn8vm\" (UniqueName: \"kubernetes.io/projected/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-kube-api-access-fn8vm\") pod \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") "
Sep 12 03:20:06.868952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.868738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-metrics-certs\") pod \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") "
Sep 12 03:20:06.868952 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.868808    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-openshift-service-ca\") pod \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\" (UID: \"cd9982a2-9bcd-407e-a18a-c77395f9bbf1\") "
Sep 12 03:20:06.869231 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.869202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd9982a2-9bcd-407e-a18a-c77395f9bbf1" (UID: "cd9982a2-9bcd-407e-a18a-c77395f9bbf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:20:06.871018 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.870984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd9982a2-9bcd-407e-a18a-c77395f9bbf1" (UID: "cd9982a2-9bcd-407e-a18a-c77395f9bbf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:20:06.871018 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.871001    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd9982a2-9bcd-407e-a18a-c77395f9bbf1" (UID: "cd9982a2-9bcd-407e-a18a-c77395f9bbf1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:20:06.871236 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.871128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-kube-api-access-fn8vm" (OuterVolumeSpecName: "kube-api-access-fn8vm") pod "cd9982a2-9bcd-407e-a18a-c77395f9bbf1" (UID: "cd9982a2-9bcd-407e-a18a-c77395f9bbf1"). InnerVolumeSpecName "kube-api-access-fn8vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:20:06.969880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.969831    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-metrics-certs\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:20:06.969880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.969870    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-openshift-service-ca\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:20:06.969880 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.969882    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-maas-api-tls\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:20:06.970185 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:06.969894    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn8vm\" (UniqueName: \"kubernetes.io/projected/cd9982a2-9bcd-407e-a18a-c77395f9bbf1-kube-api-access-fn8vm\") on node \"ip-10-0-130-57.ec2.internal\" DevicePath \"\""
Sep 12 03:20:07.499809 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.499771    2579 generic.go:356] "Generic (PLEG): container finished" podID="cd9982a2-9bcd-407e-a18a-c77395f9bbf1" containerID="d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0" exitCode=0
Sep 12 03:20:07.500000 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.499844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"
Sep 12 03:20:07.500000 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.499861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj" event={"ID":"cd9982a2-9bcd-407e-a18a-c77395f9bbf1","Type":"ContainerDied","Data":"d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0"}
Sep 12 03:20:07.500000 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.499933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj" event={"ID":"cd9982a2-9bcd-407e-a18a-c77395f9bbf1","Type":"ContainerDied","Data":"5f48f86c417d5883a06098babad5b6120135fd887b64042bd4297f5449c24499"}
Sep 12 03:20:07.500000 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.499962    2579 scope.go:117] "RemoveContainer" containerID="d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0"
Sep 12 03:20:07.509606 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.509583    2579 scope.go:117] "RemoveContainer" containerID="d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0"
Sep 12 03:20:07.509963 ip-10-0-130-57 kubenswrapper[2579]: E0912 03:20:07.509942    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0\": container with ID starting with d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0 not found: ID does not exist" containerID="d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0"
Sep 12 03:20:07.510023 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.509974    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0"} err="failed to get container status \"d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0\": rpc error: code = NotFound desc = could not find container \"d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0\": container with ID starting with d8d25c23ab238e6d76a6856488a8c8549d8931df51ecb1925fd7a86e80093bd0 not found: ID does not exist"
Sep 12 03:20:07.517986 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.517938    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"]
Sep 12 03:20:07.521897 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:07.521865    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-svvhj"]
Sep 12 03:20:09.250532 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:09.250492    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9982a2-9bcd-407e-a18a-c77395f9bbf1" path="/var/lib/kubelet/pods/cd9982a2-9bcd-407e-a18a-c77395f9bbf1/volumes"
Sep 12 03:20:13.438378 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:13.438340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:20:21.042744 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:21.042695    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9dt67"]
Sep 12 03:20:31.045829 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:31.045796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-6rhcs_b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0/copy-manifests/0.log"
Sep 12 03:20:31.052426 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:31.052400    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-6rhcs_b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0/manager/0.log"
Sep 12 03:20:31.652823 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:31.652788    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75df99c77c-hqdzj_91953934-f3e8-4e89-8dcd-18cd0b989c24/maas-api/0.log"
Sep 12 03:20:31.875142 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:31.875085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819715-7g4nm_f7bf21c7-dbae-46ec-a227-c4b90970f43a/cleanup/0.log"
Sep 12 03:20:33.049737 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:33.049698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-799sf_f86ca843-b774-486d-9bbf-37a79b279890/manager/0.log"
Sep 12 03:20:33.294671 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:33.294635    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-p8f6s_f00cad22-6379-4e30-843f-ce726ab8f488/manager/0.log"
Sep 12 03:20:33.414777 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:33.414690    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9dt67_1539d005-3280-49e1-b3f8-db9752ac3bf9/limitador/0.log"
Sep 12 03:20:34.255530 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:34.255497    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-gw4sv_83f0e59c-c794-432d-a95d-de3483af1be3/kube-auth-proxy/0.log"
Sep 12 03:20:35.229621 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:35.229592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-scg64_acd0dcd8-7df4-414b-ab2c-06cef1a94c6f/main/0.log"
Sep 12 03:20:35.836766 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:35.836730    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nf8k2_de080106-f89d-4e28-8754-a85c0e8baef4/main/0.log"
Sep 12 03:20:41.725661 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.725611    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vchk2/must-gather-ctmvj"]
Sep 12 03:20:41.726249 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.726228    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9982a2-9bcd-407e-a18a-c77395f9bbf1" containerName="maas-api"
Sep 12 03:20:41.726305 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.726254    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9982a2-9bcd-407e-a18a-c77395f9bbf1" containerName="maas-api"
Sep 12 03:20:41.726363 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.726353    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9982a2-9bcd-407e-a18a-c77395f9bbf1" containerName="maas-api"
Sep 12 03:20:41.729924 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.729901    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:41.732582 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.732551    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vchk2\"/\"kube-root-ca.crt\""
Sep 12 03:20:41.732582 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.732567    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vchk2\"/\"openshift-service-ca.crt\""
Sep 12 03:20:41.733436 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.733422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vchk2\"/\"default-dockercfg-brcb4\""
Sep 12 03:20:41.736456 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.736428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vchk2/must-gather-ctmvj"]
Sep 12 03:20:41.810007 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.809965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d-must-gather-output\") pod \"must-gather-ctmvj\" (UID: \"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d\") " pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:41.810224 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.810161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87r8\" (UniqueName: \"kubernetes.io/projected/bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d-kube-api-access-b87r8\") pod \"must-gather-ctmvj\" (UID: \"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d\") " pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:41.910926 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.910883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d-must-gather-output\") pod \"must-gather-ctmvj\" (UID: \"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d\") " pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:41.911135 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.911009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87r8\" (UniqueName: \"kubernetes.io/projected/bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d-kube-api-access-b87r8\") pod \"must-gather-ctmvj\" (UID: \"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d\") " pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:41.911268 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.911243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d-must-gather-output\") pod \"must-gather-ctmvj\" (UID: \"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d\") " pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:41.919498 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:41.919461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87r8\" (UniqueName: \"kubernetes.io/projected/bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d-kube-api-access-b87r8\") pod \"must-gather-ctmvj\" (UID: \"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d\") " pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:42.040915 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:42.040804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vchk2/must-gather-ctmvj"
Sep 12 03:20:42.316383 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:42.316354    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vchk2/must-gather-ctmvj"]
Sep 12 03:20:42.318509 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:20:42.318471    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcfbfba8_9ea2_49ce_b9f0_3479ba815d8d.slice/crio-56ae6bcb00abc268816d7d3e2d22313eb42c16a84f94f54c121ec5774f151ce7 WatchSource:0}: Error finding container 56ae6bcb00abc268816d7d3e2d22313eb42c16a84f94f54c121ec5774f151ce7: Status 404 returned error can't find the container with id 56ae6bcb00abc268816d7d3e2d22313eb42c16a84f94f54c121ec5774f151ce7
Sep 12 03:20:42.692133 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:42.692063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vchk2/must-gather-ctmvj" event={"ID":"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d","Type":"ContainerStarted","Data":"56ae6bcb00abc268816d7d3e2d22313eb42c16a84f94f54c121ec5774f151ce7"}
Sep 12 03:20:43.703365 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:43.703323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vchk2/must-gather-ctmvj" event={"ID":"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d","Type":"ContainerStarted","Data":"e04e5e6292a798ba56bd042651371857509ebcb1b97e95b56728d223426d8c33"}
Sep 12 03:20:43.703982 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:43.703955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vchk2/must-gather-ctmvj" event={"ID":"bcfbfba8-9ea2-49ce-b9f0-3479ba815d8d","Type":"ContainerStarted","Data":"093f4d68bf99e900d6665484659538e4f0ce2617e922821be5a026d47dfcc57d"}
Sep 12 03:20:43.720776 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:43.720718    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vchk2/must-gather-ctmvj" podStartSLOduration=2.144680556 podStartE2EDuration="2.720696767s" totalImagesPullingTime="576.016211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:20:43.718563042 +0000 UTC m=+2682.986596723" watchObservedRunningTime="2026-09-12 03:20:43.720696767 +0000 UTC m=+2682.988730413"
Sep 12 03:20:44.570487 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:44.570447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kw4fm_ff92f6c9-f9cb-4d8d-9056-c160e2de66cc/global-pull-secret-syncer/0.log"
Sep 12 03:20:44.693793 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:44.693751    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fmp4l_464e05fa-e1e9-4da2-bf1c-13d879fd17e5/konnectivity-agent/0.log"
Sep 12 03:20:44.770123 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:44.770067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-57.ec2.internal_22695f442950d95e8040db7f0dc4719a/haproxy/0.log"
Sep 12 03:20:49.768665 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:49.768628    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-799sf_f86ca843-b774-486d-9bbf-37a79b279890/manager/0.log"
Sep 12 03:20:49.861443 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:49.861405    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-p8f6s_f00cad22-6379-4e30-843f-ce726ab8f488/manager/0.log"
Sep 12 03:20:49.886168 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:49.886137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9dt67_1539d005-3280-49e1-b3f8-db9752ac3bf9/limitador/0.log"
Sep 12 03:20:51.725120 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:51.725048    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mwwbr_c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4/kube-state-metrics/0.log"
Sep 12 03:20:51.742926 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:51.742285    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mwwbr_c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4/kube-rbac-proxy-main/0.log"
Sep 12 03:20:51.762683 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:51.762644    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mwwbr_c3cc48e0-5a3b-4ad3-9d31-a3d750b9ccb4/kube-rbac-proxy-self/0.log"
Sep 12 03:20:51.790613 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:51.790575    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d4948d966-twqrk_84111875-128a-40db-9be5-d15f57f18175/metrics-server/0.log"
Sep 12 03:20:51.816951 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:51.816851    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-5tdvf_204e049e-66fe-4b59-b5d1-bbdc9b3a35f3/monitoring-plugin/0.log"
Sep 12 03:20:52.002649 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.002618    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvqz6_a0e5173a-6ced-49c0-8c95-0fdaecf85e42/node-exporter/0.log"
Sep 12 03:20:52.025733 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.025706    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvqz6_a0e5173a-6ced-49c0-8c95-0fdaecf85e42/kube-rbac-proxy/0.log"
Sep 12 03:20:52.048882 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.048857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvqz6_a0e5173a-6ced-49c0-8c95-0fdaecf85e42/init-textfile/0.log"
Sep 12 03:20:52.076159 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.076053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4td4t_269ee6aa-9c67-42a7-af13-617886fb4ad2/kube-rbac-proxy-main/0.log"
Sep 12 03:20:52.099767 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.099735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4td4t_269ee6aa-9c67-42a7-af13-617886fb4ad2/kube-rbac-proxy-self/0.log"
Sep 12 03:20:52.120948 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.120912    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-4td4t_269ee6aa-9c67-42a7-af13-617886fb4ad2/openshift-state-metrics/0.log"
Sep 12 03:20:52.203381 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.203284    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ce6e596-cf48-42e5-bd2b-39c90e973d61/prometheus/0.log"
Sep 12 03:20:52.225011 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.224977    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ce6e596-cf48-42e5-bd2b-39c90e973d61/config-reloader/0.log"
Sep 12 03:20:52.281579 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.281544    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ce6e596-cf48-42e5-bd2b-39c90e973d61/thanos-sidecar/0.log"
Sep 12 03:20:52.298849 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.298810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ce6e596-cf48-42e5-bd2b-39c90e973d61/kube-rbac-proxy-web/0.log"
Sep 12 03:20:52.326728 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.326632    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ce6e596-cf48-42e5-bd2b-39c90e973d61/kube-rbac-proxy/0.log"
Sep 12 03:20:52.352811 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.352785    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ce6e596-cf48-42e5-bd2b-39c90e973d61/kube-rbac-proxy-thanos/0.log"
Sep 12 03:20:52.376027 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:52.375996    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ce6e596-cf48-42e5-bd2b-39c90e973d61/init-config-reloader/0.log"
Sep 12 03:20:54.275470 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:54.275421    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-tm7d6_eee61980-927f-493a-8e93-fafe1447d816/networking-console-plugin/0.log"
Sep 12 03:20:55.863194 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.863143    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"]
Sep 12 03:20:55.867801 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.867767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:55.874399 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.874359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"]
Sep 12 03:20:55.996066 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.996032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-proc\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:55.996380 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.996359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75j4\" (UniqueName: \"kubernetes.io/projected/73f23774-89fe-4789-9c3e-63c6fb067d40-kube-api-access-r75j4\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:55.996467 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.996407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-podres\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:55.996467 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.996439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-lib-modules\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:55.996574 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:55.996516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-sys\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.099512 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.099468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-proc\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.099938 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.099917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75j4\" (UniqueName: \"kubernetes.io/projected/73f23774-89fe-4789-9c3e-63c6fb067d40-kube-api-access-r75j4\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.100116 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.100079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-podres\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.100244 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.100230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-lib-modules\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.100407 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.100389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-sys\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.103610 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.099811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-proc\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.103610 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.100577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-lib-modules\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.103610 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.100593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-sys\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.103610 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.100582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73f23774-89fe-4789-9c3e-63c6fb067d40-podres\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.112286 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.112243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75j4\" (UniqueName: \"kubernetes.io/projected/73f23774-89fe-4789-9c3e-63c6fb067d40-kube-api-access-r75j4\") pod \"perf-node-gather-daemonset-xnd2c\" (UID: \"73f23774-89fe-4789-9c3e-63c6fb067d40\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.185884 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.185833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:56.366321 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.366215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-xqh5z_3e9b3ae8-9baa-4efb-b8de-01b627a7505a/volume-data-source-validator/0.log"
Sep 12 03:20:56.503734 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.502170    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"]
Sep 12 03:20:56.506182 ip-10-0-130-57 kubenswrapper[2579]: W0912 03:20:56.506142    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod73f23774_89fe_4789_9c3e_63c6fb067d40.slice/crio-dc90f8b286b7818b4b6ebc375cccdc889a3266bb210bfca51b0cdd75a010041d WatchSource:0}: Error finding container dc90f8b286b7818b4b6ebc375cccdc889a3266bb210bfca51b0cdd75a010041d: Status 404 returned error can't find the container with id dc90f8b286b7818b4b6ebc375cccdc889a3266bb210bfca51b0cdd75a010041d
Sep 12 03:20:56.805805 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.805697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c" event={"ID":"73f23774-89fe-4789-9c3e-63c6fb067d40","Type":"ContainerStarted","Data":"1d3d5f7f9aed1f95911b22acef106f0dc06608ef226c131ad02a79567864665c"}
Sep 12 03:20:56.805805 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.805768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c" event={"ID":"73f23774-89fe-4789-9c3e-63c6fb067d40","Type":"ContainerStarted","Data":"dc90f8b286b7818b4b6ebc375cccdc889a3266bb210bfca51b0cdd75a010041d"}
Sep 12 03:20:56.806052 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:56.805938    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:20:57.525473 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:57.525436    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6znn9_4cb403fd-1ca8-4352-a7c7-ff62805c099c/dns/0.log"
Sep 12 03:20:57.546176 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:57.546137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6znn9_4cb403fd-1ca8-4352-a7c7-ff62805c099c/kube-rbac-proxy/0.log"
Sep 12 03:20:57.655080 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:57.655055    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7flt6_b0f02bda-791d-4340-a758-2269e6c97bf4/dns-node-resolver/0.log"
Sep 12 03:20:58.240059 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:58.240020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-85lmg_3b80d30f-bdd5-486d-a9e0-fe60b4a4e0de/node-ca/0.log"
Sep 12 03:20:59.666115 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:20:59.666069    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-gw4sv_83f0e59c-c794-432d-a95d-de3483af1be3/kube-auth-proxy/0.log"
Sep 12 03:21:00.695176 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:00.695147    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hrs62_7dd1fe03-285d-4060-9a10-10e9c46b5566/serve-healthcheck-canary/0.log"
Sep 12 03:21:01.223711 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:01.223670    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-57tmp_4344e4db-5901-4556-916e-c38e0ca4f7a6/insights-operator/0.log"
Sep 12 03:21:01.227706 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:01.227670    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-57tmp_4344e4db-5901-4556-916e-c38e0ca4f7a6/insights-operator/1.log"
Sep 12 03:21:01.249029 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:01.248993    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9crmx_40cb9ec2-d376-4703-a245-ca7c5cd87443/kube-rbac-proxy/0.log"
Sep 12 03:21:01.271144 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:01.271083    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9crmx_40cb9ec2-d376-4703-a245-ca7c5cd87443/exporter/0.log"
Sep 12 03:21:01.292406 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:01.292366    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9crmx_40cb9ec2-d376-4703-a245-ca7c5cd87443/extractor/0.log"
Sep 12 03:21:02.825950 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:02.825915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c"
Sep 12 03:21:02.843767 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:02.843710    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-xnd2c" podStartSLOduration=7.8436910189999995 podStartE2EDuration="7.843691019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 03:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:20:56.823368046 +0000 UTC m=+2696.091401717" watchObservedRunningTime="2026-09-12 03:21:02.843691019 +0000 UTC m=+2702.111724751"
Sep 12 03:21:04.084754 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:04.084676    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-6rhcs_b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0/manager/0.log"
Sep 12 03:21:04.105727 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:04.105693    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-6rhcs_b6c5ff9b-074d-4c8f-a0f5-b2d52b4e7ea0/copy-manifests/0.log"
Sep 12 03:21:05.545536 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:05.545499    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6645c7894d-xt89l_925b343b-f03a-4e31-b4f8-df7cdf1a4e1b/manager/0.log"
Sep 12 03:21:05.571461 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:05.571422    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vv8wg_d49bfe09-3647-42ec-a63c-35c0b592972c/openshift-lws-operator/0.log"
Sep 12 03:21:07.449385 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:07.449351    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-gw4sv_83f0e59c-c794-432d-a95d-de3483af1be3/kube-auth-proxy/0.log"
Sep 12 03:21:09.623057 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:09.623008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-gw4sv_83f0e59c-c794-432d-a95d-de3483af1be3/kube-auth-proxy/0.log"
Sep 12 03:21:11.076037 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:11.076002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-s65df_c7f29df3-0ad6-475c-b3ed-a60fb4824f5d/migrator/0.log"
Sep 12 03:21:11.097523 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:11.097491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-s65df_c7f29df3-0ad6-475c-b3ed-a60fb4824f5d/graceful-termination/0.log"
Sep 12 03:21:11.503331 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:11.503295    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-brgwx_9fd11876-ba7c-415c-8b18-c450d94fa012/kube-storage-version-migrator-operator/1.log"
Sep 12 03:21:11.505211 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:11.505182    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-brgwx_9fd11876-ba7c-415c-8b18-c450d94fa012/kube-storage-version-migrator-operator/0.log"
Sep 12 03:21:12.624519 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.624488    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sp86_11c7b4b6-0c5e-41af-9e15-d5736881babe/kube-multus-additional-cni-plugins/0.log"
Sep 12 03:21:12.644019 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.643986    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sp86_11c7b4b6-0c5e-41af-9e15-d5736881babe/egress-router-binary-copy/0.log"
Sep 12 03:21:12.663455 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.663422    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sp86_11c7b4b6-0c5e-41af-9e15-d5736881babe/cni-plugins/0.log"
Sep 12 03:21:12.683950 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.683917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sp86_11c7b4b6-0c5e-41af-9e15-d5736881babe/bond-cni-plugin/0.log"
Sep 12 03:21:12.703426 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.703393    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sp86_11c7b4b6-0c5e-41af-9e15-d5736881babe/routeoverride-cni/0.log"
Sep 12 03:21:12.722156 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.722126    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sp86_11c7b4b6-0c5e-41af-9e15-d5736881babe/whereabouts-cni-bincopy/0.log"
Sep 12 03:21:12.741988 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.741954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7sp86_11c7b4b6-0c5e-41af-9e15-d5736881babe/whereabouts-cni/0.log"
Sep 12 03:21:12.990462 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:12.990430    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-plcc8_1c526b2d-2382-47f3-84f1-4dfcc2a0ee8b/kube-multus/0.log"
Sep 12 03:21:13.048664 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:13.048636    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gncjn_c0eba23f-369a-4cec-954d-d152d68cc4bd/network-metrics-daemon/0.log"
Sep 12 03:21:13.062275 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:13.062243    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gncjn_c0eba23f-369a-4cec-954d-d152d68cc4bd/kube-rbac-proxy/0.log"
Sep 12 03:21:14.194201 ip-10-0-130-57 kubenswrapper[2579]: I0912 03:21:14.194153    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grws5_a38160e4-6c6f-432d-8011-0f7472ed7d9c/ovn-controller/0.log"