Sep 11 15:55:02.218004 ip-10-0-129-137 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 15:55:02.600035 ip-10-0-129-137 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 15:55:02.600035 ip-10-0-129-137 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 15:55:02.600035 ip-10-0-129-137 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 15:55:02.600035 ip-10-0-129-137 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 15:55:02.600035 ip-10-0-129-137 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 15:55:02.600924 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.600162    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 15:55:02.605894 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.605858    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:02.606703 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606505    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:02.606703 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606700    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:02.606703 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606705    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:02.606703 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606709    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606712    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606716    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606720    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606724    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606726    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606729    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606732    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606735    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606738    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606741    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606744    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606746    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606749    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606752    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606755    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606757    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606761    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606764    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606766    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:02.606886 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606769    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606772    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606774    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606777    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606785    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606788    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606791    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606793    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606796    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606799    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606802    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606805    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606807    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606811    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606814    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606817    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606820    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606826    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606829    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606832    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:02.607406 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606835    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606838    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606843    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606847    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606849    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606854    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606859    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606862    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606866    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606869    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606873    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606876    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606879    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606882    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606885    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606888    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606891    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606894    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606897    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:02.607928 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606900    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606904    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606908    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606911    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606914    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606917    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606923    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606928    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606931    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606934    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606937    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606940    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606943    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606946    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606949    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606952    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606954    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606957    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606960    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606962    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:02.608384 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606965    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606968    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606971    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606973    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606976    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606979    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606982    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606984    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606987    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606990    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606993    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606995    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.606998    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607000    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607003    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607006    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607770    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607778    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607782    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607785    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:02.608887 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607788    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607790    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607793    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607796    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607799    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607803    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607806    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607808    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607811    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607814    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607816    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607819    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607822    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607824    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607828    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607830    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607833    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607836    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607840    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:02.609357 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607845    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607848    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607851    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607854    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607857    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607860    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607863    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607865    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607869    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607873    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607876    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607880    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607883    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607886    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607889    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607892    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607896    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607898    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607901    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607904    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:02.609865 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607906    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607909    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607912    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607915    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607917    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607920    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607923    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607927    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607929    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607932    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607935    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607938    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607940    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607943    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607945    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607949    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607951    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607954    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607956    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607959    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:02.610360 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607961    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607964    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607966    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607969    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607973    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607976    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607978    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607981    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607984    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607987    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607989    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607992    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607995    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.607997    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608000    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608004    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608006    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608009    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608011    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608014    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:02.610873 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608017    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608020    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608023    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608025    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608029    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608033    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608036    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608039    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608042    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608045    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608047    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608050    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608053    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608056    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608059    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608063    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608467    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608472    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608476    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:02.611368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608479    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608482    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608485    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608488    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608491    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608494    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608497    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608499    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608502    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608505    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608508    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608511    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608514    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608517    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608520    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608523    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608526    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608529    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608531    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608534    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:02.611854 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608537    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608540    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608544    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608547    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608550    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608552    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608555    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608560    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608563    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608566    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608569    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608572    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608574    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608577    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608579    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608582    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608584    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608587    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608590    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:02.612352 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608592    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608595    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608597    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608601    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608605    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608607    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608610    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608613    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608616    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608618    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608621    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608623    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608626    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608629    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608631    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608634    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608636    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608639    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608642    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608644    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:02.612840 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608665    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608669    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608671    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608674    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608677    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608680    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608683    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608685    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608688    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608691    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608694    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608696    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608699    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608702    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608704    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608707    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608710    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608712    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608715    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608718    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:02.613426 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608726    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608732    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608735    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608738    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608741    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608743    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608746    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608748    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608751    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608754    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608757    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608759    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608763    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608766    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608769    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608772    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.608774    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608848    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608884    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608892    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608897    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 15:55:02.613938 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608901    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608905    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608909    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608914    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608918    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608921    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608924    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608928    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608931    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608934    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608937    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608941    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608945    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608948    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608952    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608955    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608958    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608962    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608965    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608969    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608973    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608976    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608979    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608983    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 15:55:02.614458 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608987    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608990    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608993    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.608996    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609000    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609003    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609006    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609009    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609012    2559 flags.go:64] FLAG: --enable-server="true"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609015    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609020    2559 flags.go:64] FLAG: --event-burst="100"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609023    2559 flags.go:64] FLAG: --event-qps="50"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609026    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609029    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609032    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609036    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609039    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609042    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609045    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609048    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609051    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609054    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609057    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609060    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609063    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 15:55:02.615099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609066    2559 flags.go:64] FLAG: --feature-gates=""
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609070    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609073    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609076    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609080    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609083    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609086    2559 flags.go:64] FLAG: --help="false"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609089    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609092    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609096    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609099    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609102    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609105    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609109    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609112    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609115    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609118    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609121    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609124    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609126    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609129    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609132    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609135    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609138    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 15:55:02.615736 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609141    2559 flags.go:64] FLAG: --lock-file=""
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609144    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609147    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609151    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609156    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609160    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609163    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609166    2559 flags.go:64] FLAG: --logging-format="text"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609169    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609172    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609175    2559 flags.go:64] FLAG: --manifest-url=""
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609178    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609184    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609188    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609192    2559 flags.go:64] FLAG: --max-pods="110"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609196    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609199    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609202    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609205    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609208    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609212    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609215    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609221    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609225    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609228    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 15:55:02.616371 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609231    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609234    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609239    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609242    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609245    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609248    2559 flags.go:64] FLAG: --port="10250"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609251    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609254    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b18f6c567867c346"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609258    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609261    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609264    2559 flags.go:64] FLAG: --register-node="true"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609267    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609270    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609274    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609277    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609280    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609283    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609286    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609289    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609293    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609296    2559 flags.go:64] FLAG: --runonce="false"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609299    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609302    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609305    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609308    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 15:55:02.617038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609311    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609314    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609317    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609320    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609324    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609327    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609330    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609333    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609336    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609339    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609342    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609347    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609349    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609352    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609356    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609359    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609362    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609365    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609368    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609371    2559 flags.go:64] FLAG: --v="2"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609376    2559 flags.go:64] FLAG: --version="false"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609380    2559 flags.go:64] FLAG: --vmodule=""
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609384    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.609388    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609519    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:02.617672 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609523    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609528    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609531    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609534    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609539    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609541    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609544    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609547    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609550    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609553    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609556    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609560    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609564    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609567    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609570    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609573    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609575    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609580    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609583    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:02.618300 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609585    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609588    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609591    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609594    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609597    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609600    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609603    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609606    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609609    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609612    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609615    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609618    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609620    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609623    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609627    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609630    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609633    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609637    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609641    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609644    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:02.618830 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609662    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609666    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609670    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609674    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609677    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609679    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609682    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609685    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609688    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609690    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609693    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609696    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609698    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609701    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609704    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609707    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609709    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609712    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609715    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609717    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:02.619327 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609720    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609723    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609725    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609730    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609734    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609737    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609741    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609744    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609747    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609751    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609754    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609756    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609759    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609762    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609765    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609767    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609770    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609773    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609776    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:02.619843 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609778    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609781    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609784    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609786    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609789    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609792    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609794    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609797    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609799    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609802    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609804    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609808    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609812    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609815    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609818    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609820    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609823    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609826    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609828    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:02.620325 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.609832    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.610346    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.618014    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.618034    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.618671    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.618679    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618745    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618752    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618756    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618760    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618763    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618766    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618770    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618773    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:02.620821 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618776    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618779    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618782    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618785    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618787    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618790    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618793    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618796    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618798    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618801    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618804    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618807    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618810    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618812    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618815    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618818    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618820    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618823    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618825    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618828    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:02.621177 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618831    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618835    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618838    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618841    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618844    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618847    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618849    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618853    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618855    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618858    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618861    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618863    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618867    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618872    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618875    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618877    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618880    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618883    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618885    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:02.621686 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618888    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618891    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618893    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618896    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618898    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618901    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618903    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618906    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618909    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618912    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618914    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618917    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618920    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618924    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618927    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618930    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618933    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618936    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618939    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618942    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:02.622152 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618945    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618948    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618950    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618954    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618958    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618961    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618964    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618967    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618970    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618973    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618975    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618978    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618981    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618984    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618987    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618990    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618993    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618995    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.618998    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:02.622765 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619001    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619003    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619006    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619009    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619012    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619014    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619017    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619021    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619023    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619026    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619029    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619032    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619035    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.619041    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619164    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:02.623318 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619168    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619172    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619174    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619177    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619180    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619183    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619185    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619188    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619190    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619193    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619197    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619199    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619202    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619204    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619207    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619211    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619215    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619217    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619220    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619223    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:02.623739 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619226    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619229    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619231    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619234    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619237    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619240    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619244    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619247    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619250    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619252    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619255    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619258    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619260    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619263    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619265    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619268    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619271    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619274    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619276    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619279    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:02.624285 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619281    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619284    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619287    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619289    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619292    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619294    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619297    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619300    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619302    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619305    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619308    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619310    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619313    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619315    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619318    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619320    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619324    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619327    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619330    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619332    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:02.624962 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619336    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619339    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619342    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619346    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619349    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619352    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619355    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619358    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619361    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619365    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619368    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619370    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619373    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619376    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619379    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619382    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619384    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619387    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619389    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:02.625737 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619392    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619394    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619397    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619400    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619402    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619405    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619408    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619411    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619413    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619416    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619419    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619421    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619424    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619427    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619430    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619433    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619435    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619438    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:02.626302 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:02.619441    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:02.626863 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.619446    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 15:55:02.626863 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.619598    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 15:55:02.626863 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.622177    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 15:55:02.626863 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.623060    2559 server.go:1019] "Starting client certificate rotation"
Sep 11 15:55:02.626863 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.623168    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 15:55:02.626863 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.623222    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 15:55:02.626863 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.626401    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 15:55:02.628336 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.628315    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 15:55:02.637896 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.637876    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 11 15:55:02.642887 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.642870    2559 log.go:25] "Validated CRI v1 image API"
Sep 11 15:55:02.643719 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.643701    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 15:55:02.647293 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.647273    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9f94a767-740c-4132-bd13-02a87ea0c720:/dev/nvme0n1p4 c5a14698-4ca3-4817-b43e-ef6f51dcb6e9:/dev/nvme0n1p3]
Sep 11 15:55:02.647353 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.647293    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 15:55:02.652661 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.652517    2559 manager.go:216] Machine: {Timestamp:2026-09-11 15:55:02.651356497 +0000 UTC m=+0.316759841 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106089 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b10950fb9aa95faee37dcd13c3759 SystemUUID:ec2b1095-0fb9-aa95-faee-37dcd13c3759 BootID:ea77f086-e9aa-4b30-9ba0-154d2f97086d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:cc:7c:ad:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:cc:7c:ad:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:7d:9f:b6:42:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 15:55:02.652661 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.652636    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 15:55:02.652788 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.652747    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 15:55:02.653851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.653833    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 15:55:02.655695 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.655661    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 15:55:02.655855 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.655698    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 15:55:02.657316 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.657303    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 15:55:02.657364 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.657318    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 15:55:02.657364 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.657327    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 15:55:02.657905 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.657895    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 15:55:02.658700 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.658690    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 15:55:02.662435 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.662423    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 15:55:02.662569 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.662559    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 15:55:02.662600 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.662591    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 15:55:02.662627 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.662608    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 15:55:02.662627 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.662624    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 15:55:02.662688 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.662634    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 11 15:55:02.662688 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.662644    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 15:55:02.663081 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.663069    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 15:55:02.663120 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.663089    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 15:55:02.665168 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.665152    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 15:55:02.667291 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.667277    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 15:55:02.667330 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.667318    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 15:55:02.669424 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669408    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 15:55:02.669499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669430    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 15:55:02.669499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669437    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 15:55:02.669499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669442    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 15:55:02.669499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669451    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 15:55:02.669499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669457    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 15:55:02.669499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669470    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 15:55:02.669667 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669503    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 15:55:02.669667 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669515    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 15:55:02.669667 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669529    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 15:55:02.669667 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669538    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 15:55:02.669667 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.669550    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 15:55:02.670163 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.670151    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 15:55:02.670192 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.670167    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 15:55:02.674390 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.674371    2559 server.go:1268] "Started kubelet"
Sep 11 15:55:02.674491 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.674451    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 15:55:02.674567 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.674525    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 15:55:02.674616 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.674440    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 15:55:02.674989 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.674964    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 15:55:02.675206 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.675182    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 15:55:02.675621 ip-10-0-129-137 systemd[1]: Started Kubernetes Kubelet.
Sep 11 15:55:02.680461 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.679342    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 15:55:02.680632 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.675323    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 15:55:02.682079 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.681556    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xb9l8"
Sep 11 15:55:02.683523 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.683505    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 15:55:02.683802 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.683778    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 15:55:02.684436 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.684416    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xb9l8"
Sep 11 15:55:02.689232 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.689215    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 15:55:02.689788 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.689768    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 15:55:02.690008 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.689955    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:02.690451 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.690439    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 15:55:02.690514 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.690454    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 15:55:02.690514 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.690440    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 15:55:02.690846 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.690820    2559 factory.go:55] Registering systemd factory
Sep 11 15:55:02.690915 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.690863    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 11 15:55:02.691116 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.690240    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18d44f435a9989c1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-09-11 15:55:02.674344385 +0000 UTC m=+0.339747708,LastTimestamp:2026-09-11 15:55:02.674344385 +0000 UTC m=+0.339747708,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}"
Sep 11 15:55:02.691378 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.691362    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 15:55:02.691378 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.691375    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 15:55:02.691637 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.691620    2559 factory.go:153] Registering CRI-O factory
Sep 11 15:55:02.691702 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.691642    2559 factory.go:223] Registration of the crio container factory successfully
Sep 11 15:55:02.691746 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.691714    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 15:55:02.691746 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.691738    2559 factory.go:103] Registering Raw factory
Sep 11 15:55:02.691840 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.691756    2559 manager.go:1196] Started watching for new ooms in manager
Sep 11 15:55:02.692277 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.692244    2559 manager.go:349] Starting recovery of all containers
Sep 11 15:55:02.699716 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.699508    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:02.702423 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.702402    2559 manager.go:354] Recovery completed
Sep 11 15:55:02.703841 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.703817    2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.708973 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.708958    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:02.710795 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.710777    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:02.710889 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.710823    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:02.710889 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.710838    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:02.711534 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.711521    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 15:55:02.711608 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.711534    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 15:55:02.711608 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.711539    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:02.711608 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.711554    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 15:55:02.713375 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.713356    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:02.713444 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.713388    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:02.713444 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.713403    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:02.713859 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.713844    2559 policy_none.go:49] "None policy: Start"
Sep 11 15:55:02.713925 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.713864    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 15:55:02.713925 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.713877    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 15:55:02.715915 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.715902    2559 policy_none.go:47] "Start"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.751368    2559 manager.go:337] "Starting Device Plugin manager"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.751436    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.751451    2559 server.go:87] "Starting device plugin registration server"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.751923    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.751938    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.752052    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.752130    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.752139    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.752676    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 15:55:02.766046 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.752723    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:02.784842 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.784807    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 15:55:02.785976 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.785954    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 15:55:02.786064 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.786019    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 15:55:02.786064 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.786054    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 15:55:02.786179 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.786163    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 15:55:02.788461 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.788439    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:02.852729 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.852578    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:02.853615 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.853590    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:02.853748 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.853625    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:02.853748 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.853640    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:02.853748 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.853683    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.862469 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.862448    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.862608 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.862475    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-137.ec2.internal\": node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:02.884954 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.884917    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:02.886938 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.886905    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"]
Sep 11 15:55:02.887042 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.887006    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:02.888095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.888074    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:02.888188 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.888111    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:02.888188 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.888123    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:02.890546 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.890526    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:02.890672 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.890643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.890735 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.890706    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:02.891554 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.891535    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67c01a9ad6d35b61be659f6d32e178b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"67c01a9ad6d35b61be659f6d32e178b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.891630 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.891559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fa926f4c1f4ea217ab8dc7d0d74214a4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"fa926f4c1f4ea217ab8dc7d0d74214a4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.891630 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.891581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67c01a9ad6d35b61be659f6d32e178b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"67c01a9ad6d35b61be659f6d32e178b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.891869 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.891855    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:02.891905 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.891882    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:02.891905 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.891891    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:02.893926 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.893914    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:02.893978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.893937    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:02.894014 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.893970    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:02.894117 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.894107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.894149 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.894130    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:02.894832 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.894817    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:02.894903 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.894845    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:02.894903 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.894855    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:02.928535 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.928513    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.933125 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.933105    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.986082 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:02.986048    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:02.992640 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.992611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fa926f4c1f4ea217ab8dc7d0d74214a4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"fa926f4c1f4ea217ab8dc7d0d74214a4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.992733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.992660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67c01a9ad6d35b61be659f6d32e178b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"67c01a9ad6d35b61be659f6d32e178b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.992733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.992683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67c01a9ad6d35b61be659f6d32e178b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"67c01a9ad6d35b61be659f6d32e178b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.992830 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.992739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67c01a9ad6d35b61be659f6d32e178b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"67c01a9ad6d35b61be659f6d32e178b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.992830 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.992744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fa926f4c1f4ea217ab8dc7d0d74214a4-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"fa926f4c1f4ea217ab8dc7d0d74214a4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:02.992830 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:02.992749    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67c01a9ad6d35b61be659f6d32e178b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"67c01a9ad6d35b61be659f6d32e178b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:03.086839 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:03.086796    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:03.187683 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:03.187583    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:03.231085 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:03.231054    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:03.235844 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:03.235829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:04.529478 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:04.529447    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:04.629969 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:04.629928    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found"
Sep 11 15:55:04.725641 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.725577    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:04.731685 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.731668    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:04.748945 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.748916    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 15:55:04.750542 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.750524    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal"
Sep 11 15:55:04.761510 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.761492    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 15:55:04.864921 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.864871    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 15:55:04.865407 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.865087    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 15:55:04.865407 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.865100    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 15:55:04.865407 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.865101    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 15:55:04.905162 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.905130    2559 apiserver.go:52] "Watching apiserver"
Sep 11 15:55:04.922301 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.922269    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 15:55:04.924343 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.924319    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wl7tc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv","openshift-cluster-node-tuning-operator/tuned-k6sqh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal","openshift-multus/multus-additional-cni-plugins-7jf2w","openshift-multus/multus-bdmc8","openshift-network-operator/iptables-alerter-cjzgs","openshift-ovn-kubernetes/ovnkube-node-bfjkn","kube-system/global-pull-secret-syncer-nnlqc","kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal","openshift-dns/node-resolver-78ppt","openshift-image-registry/node-ca-lhrj7","openshift-multus/network-metrics-daemon-hspt8","openshift-network-diagnostics/network-check-target-dvfp8"]
Sep 11 15:55:04.926835 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.926816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:04.928988 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.928965    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.930971 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.930945    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 15:55:04.931092 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.931060    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.931738 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.931650    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 15:50:03 +0000 UTC" deadline="2028-06-08 13:07:25.290127888 +0000 UTC"
Sep 11 15:55:04.933282 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.931741    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15261h12m20.35839553s"
Sep 11 15:55:04.933825 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.933802    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 15:55:04.933945 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.933811    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 15:55:04.934058 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.933816    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 15:55:04.934058 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.933993    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rl7kn\""
Sep 11 15:55:04.934058 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.933854    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 15:55:04.934058 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.934043    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 15:55:04.934279 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.934131    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7rnp\""
Sep 11 15:55:04.934350 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.934331    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.935150 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.935132    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 15:55:04.935346 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.935332    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 15:55:04.935740 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.935725    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h56bx\""
Sep 11 15:55:04.936586 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.936571    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:04.937821 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937800    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.937897 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.937897 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-socket-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.937998 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-kubernetes\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.937998 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysctl-d\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.937998 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-registration-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.937998 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysconfig\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.937998 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-lib-modules\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.937998 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.937987    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-cnibin\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-systemd\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-sys\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a5b87e4-2e32-4545-947d-4482ab103ac5-agent-certs\") pod \"konnectivity-agent-wl7tc\" (UID: \"5a5b87e4-2e32-4545-947d-4482ab103ac5\") " pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a5b87e4-2e32-4545-947d-4482ab103ac5-konnectivity-ca\") pod \"konnectivity-agent-wl7tc\" (UID: \"5a5b87e4-2e32-4545-947d-4482ab103ac5\") " pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-sys-fs\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938132    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysctl-conf\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-host\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938163    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hm7\" (UniqueName: \"kubernetes.io/projected/bd2c946e-365c-4f0f-88e3-652083f474cb-kube-api-access-k6hm7\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-modprobe-d\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-system-cni-dir\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.938310 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-tuned\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxwr\" (UniqueName: \"kubernetes.io/projected/025941d9-b594-46d9-b5b6-e212db101199-kube-api-access-qfxwr\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938362    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-device-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-run\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-var-lib-kubelet\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3412bf23-2b1c-47a3-8cfe-89e18522785a-tmp\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75mcp\" (UniqueName: \"kubernetes.io/projected/3412bf23-2b1c-47a3-8cfe-89e18522785a-kube-api-access-75mcp\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938605    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b549p\""
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938617    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 15:55:04.938741 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938625    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 15:55:04.939184 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.938880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:04.939184 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.939137    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 15:55:04.940383 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.940366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 15:55:04.941299 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.941276    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:04.942986 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.942964    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fpwg7\""
Sep 11 15:55:04.943077 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.942986    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 15:55:04.943181 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.943168    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 15:55:04.943228 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.943214    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m5lk7\""
Sep 11 15:55:04.943439 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.943423    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 15:55:04.943503 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.943488    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 15:55:04.943554 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.943537    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 15:55:04.943554 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.943549    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:04.943644 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:04.943624    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:04.944820 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.944796    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 15:55:04.944893 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.944819    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 15:55:04.945006 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.944990    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h99f2\""
Sep 11 15:55:04.945189 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.945175    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 15:55:04.945752 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.945739    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:04.946819 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.946775    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 15:55:04.946819 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.946777    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 15:55:04.947921 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.947905    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 15:55:04.948002 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.947941    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:04.948214 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.948191    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 15:55:04.948505 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.948478    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 15:55:04.948609 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.948521    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 15:55:04.948882 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.948868    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pssvp\""
Sep 11 15:55:04.950848 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.950828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:04.950912 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:04.950889    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:04.953120 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.953095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:04.953220 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:04.953156    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:04.953220 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.953189    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 15:55:04.953443 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.953429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xppsj\""
Sep 11 15:55:04.953489 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.953467    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 15:55:04.953921 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.953907    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 15:55:04.998941 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:04.998913    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4brl"
Sep 11 15:55:05.007145 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.007124    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4brl"
Sep 11 15:55:05.033626 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.033597    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 15:55:05.038814 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnhm\" (UniqueName: \"kubernetes.io/projected/4cab6e56-d6b3-4161-b5ca-34d418afb13f-kube-api-access-tvnhm\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.038939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovnkube-config\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.038939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.038939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3412bf23-2b1c-47a3-8cfe-89e18522785a-tmp\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.038939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75mcp\" (UniqueName: \"kubernetes.io/projected/3412bf23-2b1c-47a3-8cfe-89e18522785a-kube-api-access-75mcp\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.038939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038894    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-cni-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.038939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.038939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-systemd-units\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-cni-bin\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-registration-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.038988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysconfig\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-lib-modules\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-host-slash\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039035    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-sys\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-node-log\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dabccca7-3201-4e1e-84d7-ef82f064128a-hosts-file\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabccca7-3201-4e1e-84d7-ef82f064128a-tmp-dir\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcp55\" (UniqueName: \"kubernetes.io/projected/dabccca7-3201-4e1e-84d7-ef82f064128a-kube-api-access-mcp55\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-cnibin\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-hostroot\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljdf4\" (UniqueName: \"kubernetes.io/projected/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-kube-api-access-ljdf4\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a5b87e4-2e32-4545-947d-4482ab103ac5-agent-certs\") pod \"konnectivity-agent-wl7tc\" (UID: \"5a5b87e4-2e32-4545-947d-4482ab103ac5\") " pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-host\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-multus-certs\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.039297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwblm\" (UniqueName: \"kubernetes.io/projected/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-kube-api-access-fwblm\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hm7\" (UniqueName: \"kubernetes.io/projected/bd2c946e-365c-4f0f-88e3-652083f474cb-kube-api-access-k6hm7\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-modprobe-d\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039292    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-systemd\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-sys\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysconfig\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-lib-modules\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-modprobe-d\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-system-cni-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-tuned\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-socket-dir-parent\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.040083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039786    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-kubelet\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039800    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cab6e56-d6b3-4161-b5ca-34d418afb13f-host\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxwr\" (UniqueName: \"kubernetes.io/projected/025941d9-b594-46d9-b5b6-e212db101199-kube-api-access-qfxwr\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-device-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-run\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-device-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.039977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-registration-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040035    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-var-lib-kubelet\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-slash\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-log-socket\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflp5\" (UniqueName: \"kubernetes.io/projected/9e9178ca-3477-4a26-b317-af72dd06fb8f-kube-api-access-rflp5\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-var-lib-kubelet\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-host\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81f8ab6-37ce-477f-9eae-90441a29d9b3-kubelet-config\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.041089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040257    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-kubelet\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040295    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-run\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-var-lib-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-etc-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040354    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-iptables-alerter-script\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-socket-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-kubernetes\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysctl-d\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-cni-binary-copy\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-conf-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040578    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkrs\" (UniqueName: \"kubernetes.io/projected/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-kube-api-access-cgkrs\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040606    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-kubernetes\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-cnibin\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-systemd\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042023 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-daemon-config\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cab6e56-d6b3-4161-b5ca-34d418afb13f-serviceca\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a5b87e4-2e32-4545-947d-4482ab103ac5-konnectivity-ca\") pod \"konnectivity-agent-wl7tc\" (UID: \"5a5b87e4-2e32-4545-947d-4482ab103ac5\") " pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-sys-fs\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysctl-conf\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-socket-dir\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-cnibin\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040913    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-systemd\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040926    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysctl-conf\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-ovn\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.040988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovnkube-script-lib\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81f8ab6-37ce-477f-9eae-90441a29d9b3-dbus\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-netns\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovn-node-metrics-cert\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-k8s-cni-cncf-io\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-cni-multus\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-system-cni-dir\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.042978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041212    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041257    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-cni-bin\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-run-netns\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-cni-netd\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041395    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-env-overrides\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a5b87e4-2e32-4545-947d-4482ab103ac5-konnectivity-ca\") pod \"konnectivity-agent-wl7tc\" (UID: \"5a5b87e4-2e32-4545-947d-4482ab103ac5\") " pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-etc-kubernetes\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd2c946e-365c-4f0f-88e3-652083f474cb-sys-fs\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-sysctl-d\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.041505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/025941d9-b594-46d9-b5b6-e212db101199-system-cni-dir\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.042095    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.042107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.042789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/025941d9-b594-46d9-b5b6-e212db101199-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.043432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3412bf23-2b1c-47a3-8cfe-89e18522785a-tmp\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.043657    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3412bf23-2b1c-47a3-8cfe-89e18522785a-etc-tuned\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.043683 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.043667    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a5b87e4-2e32-4545-947d-4482ab103ac5-agent-certs\") pod \"konnectivity-agent-wl7tc\" (UID: \"5a5b87e4-2e32-4545-947d-4482ab103ac5\") " pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:05.050804 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.050777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hm7\" (UniqueName: \"kubernetes.io/projected/bd2c946e-365c-4f0f-88e3-652083f474cb-kube-api-access-k6hm7\") pod \"aws-ebs-csi-driver-node-b9jtv\" (UID: \"bd2c946e-365c-4f0f-88e3-652083f474cb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.051450 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.051421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxwr\" (UniqueName: \"kubernetes.io/projected/025941d9-b594-46d9-b5b6-e212db101199-kube-api-access-qfxwr\") pod \"multus-additional-cni-plugins-7jf2w\" (UID: \"025941d9-b594-46d9-b5b6-e212db101199\") " pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.051450 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.051432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75mcp\" (UniqueName: \"kubernetes.io/projected/3412bf23-2b1c-47a3-8cfe-89e18522785a-kube-api-access-75mcp\") pod \"tuned-k6sqh\" (UID: \"3412bf23-2b1c-47a3-8cfe-89e18522785a\") " pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.065686 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.065662    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:05.141638 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-systemd\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.141638 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-system-cni-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-system-cni-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-socket-dir-parent\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-kubelet\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.141851 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.141850    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141858    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cab6e56-d6b3-4161-b5ca-34d418afb13f-host\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141883    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-slash\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-socket-dir-parent\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-slash\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-systemd\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.141957    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret podName:c81f8ab6-37ce-477f-9eae-90441a29d9b3 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:05.641934061 +0000 UTC m=+2.065751752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret") pod "global-pull-secret-syncer-nnlqc" (UID: "c81f8ab6-37ce-477f-9eae-90441a29d9b3") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-kubelet\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cab6e56-d6b3-4161-b5ca-34d418afb13f-host\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.141995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-log-socket\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142030    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-log-socket\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflp5\" (UniqueName: \"kubernetes.io/projected/9e9178ca-3477-4a26-b317-af72dd06fb8f-kube-api-access-rflp5\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81f8ab6-37ce-477f-9eae-90441a29d9b3-kubelet-config\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-kubelet\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-var-lib-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.142193 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142150    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-kubelet\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81f8ab6-37ce-477f-9eae-90441a29d9b3-kubelet-config\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142197    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-etc-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-var-lib-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-etc-openvswitch\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-iptables-alerter-script\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-cni-binary-copy\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-conf-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkrs\" (UniqueName: \"kubernetes.io/projected/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-kube-api-access-cgkrs\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142452    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-conf-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-daemon-config\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cab6e56-d6b3-4161-b5ca-34d418afb13f-serviceca\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-ovn\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovnkube-script-lib\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81f8ab6-37ce-477f-9eae-90441a29d9b3-dbus\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-netns\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovn-node-metrics-cert\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-k8s-cni-cncf-io\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-cni-multus\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-cni-bin\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-run-netns\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-cni-netd\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-env-overrides\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-etc-kubernetes\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnhm\" (UniqueName: \"kubernetes.io/projected/4cab6e56-d6b3-4161-b5ca-34d418afb13f-kube-api-access-tvnhm\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-cni-binary-copy\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovnkube-config\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-k8s-cni-cncf-io\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-iptables-alerter-script\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142952    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-daemon-config\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.142989    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-cni-multus\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81f8ab6-37ce-477f-9eae-90441a29d9b3-dbus\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-run-netns\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-var-lib-cni-bin\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.143939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143060    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-run-ovn\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-cni-netd\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-etc-kubernetes\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-netns\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-cni-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4cab6e56-d6b3-4161-b5ca-34d418afb13f-serviceca\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-systemd-units\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-cni-bin\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-host-slash\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-multus-cni-dir\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-node-log\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovnkube-config\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-host-cni-bin\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143561    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dabccca7-3201-4e1e-84d7-ef82f064128a-hosts-file\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-systemd-units\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143588    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e9178ca-3477-4a26-b317-af72dd06fb8f-node-log\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143588    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabccca7-3201-4e1e-84d7-ef82f064128a-tmp-dir\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dabccca7-3201-4e1e-84d7-ef82f064128a-hosts-file\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.144524 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-host-slash\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcp55\" (UniqueName: \"kubernetes.io/projected/dabccca7-3201-4e1e-84d7-ef82f064128a-kube-api-access-mcp55\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-cnibin\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-hostroot\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljdf4\" (UniqueName: \"kubernetes.io/projected/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-kube-api-access-ljdf4\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-multus-certs\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwblm\" (UniqueName: \"kubernetes.io/projected/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-kube-api-access-fwblm\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabccca7-3201-4e1e-84d7-ef82f064128a-tmp-dir\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-env-overrides\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-host-run-multus-certs\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-cnibin\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.143924    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.143940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-hostroot\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.143992    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs podName:e4a20019-f1cd-4ee9-bd36-ba1d1e50a527 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:05.643975168 +0000 UTC m=+2.067792847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs") pod "network-metrics-daemon-hspt8" (UID: "e4a20019-f1cd-4ee9-bd36-ba1d1e50a527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:05.145087 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.144305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovnkube-script-lib\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.146331 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.146311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e9178ca-3477-4a26-b317-af72dd06fb8f-ovn-node-metrics-cert\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.149137 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.149121    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:05.149194 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.149140    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:05.149194 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.149150    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5v5wh for pod openshift-network-diagnostics/network-check-target-dvfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:05.149276 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.149208    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh podName:d683e04d-faba-49bc-be34-f6e201a52d05 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:05.649194959 +0000 UTC m=+2.073012631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5wh" (UniqueName: "kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh") pod "network-check-target-dvfp8" (UID: "d683e04d-faba-49bc-be34-f6e201a52d05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:05.152031 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.152003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkrs\" (UniqueName: \"kubernetes.io/projected/430c3e42-d8d6-4937-ab4a-65c6dc9ae962-kube-api-access-cgkrs\") pod \"multus-bdmc8\" (UID: \"430c3e42-d8d6-4937-ab4a-65c6dc9ae962\") " pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.152031 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.152007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflp5\" (UniqueName: \"kubernetes.io/projected/9e9178ca-3477-4a26-b317-af72dd06fb8f-kube-api-access-rflp5\") pod \"ovnkube-node-bfjkn\" (UID: \"9e9178ca-3477-4a26-b317-af72dd06fb8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.152253 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.152073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljdf4\" (UniqueName: \"kubernetes.io/projected/b1ad41d4-ca46-4cdd-98b8-e198ba5723e3-kube-api-access-ljdf4\") pod \"iptables-alerter-cjzgs\" (UID: \"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3\") " pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.152769 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.152743    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwblm\" (UniqueName: \"kubernetes.io/projected/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-kube-api-access-fwblm\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:05.153694 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.153671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnhm\" (UniqueName: \"kubernetes.io/projected/4cab6e56-d6b3-4161-b5ca-34d418afb13f-kube-api-access-tvnhm\") pod \"node-ca-lhrj7\" (UID: \"4cab6e56-d6b3-4161-b5ca-34d418afb13f\") " pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.154177 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.154156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcp55\" (UniqueName: \"kubernetes.io/projected/dabccca7-3201-4e1e-84d7-ef82f064128a-kube-api-access-mcp55\") pod \"node-resolver-78ppt\" (UID: \"dabccca7-3201-4e1e-84d7-ef82f064128a\") " pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.159263 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.159246    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 15:55:05.258926 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.258897    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:05.271517 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.271474    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5b87e4_2e32_4545_947d_4482ab103ac5.slice/crio-3cb072cc69a148921df15b9f3966caadb414fb73545a7231deb2f37737d9ac8d WatchSource:0}: Error finding container 3cb072cc69a148921df15b9f3966caadb414fb73545a7231deb2f37737d9ac8d: Status 404 returned error can't find the container with id 3cb072cc69a148921df15b9f3966caadb414fb73545a7231deb2f37737d9ac8d
Sep 11 15:55:05.272151 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.272116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv"
Sep 11 15:55:05.280175 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.280153    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2c946e_365c_4f0f_88e3_652083f474cb.slice/crio-93fc898a4b38ef296b029836be0d27b1bc45b47a5a3544c79c5f5b8f5ef18c6f WatchSource:0}: Error finding container 93fc898a4b38ef296b029836be0d27b1bc45b47a5a3544c79c5f5b8f5ef18c6f: Status 404 returned error can't find the container with id 93fc898a4b38ef296b029836be0d27b1bc45b47a5a3544c79c5f5b8f5ef18c6f
Sep 11 15:55:05.280836 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.280819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k6sqh"
Sep 11 15:55:05.289213 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.289190    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3412bf23_2b1c_47a3_8cfe_89e18522785a.slice/crio-4d881a1c2010ca43d40b3fa03ce715cdf05b6576ebc1b1f270048c484d217408 WatchSource:0}: Error finding container 4d881a1c2010ca43d40b3fa03ce715cdf05b6576ebc1b1f270048c484d217408: Status 404 returned error can't find the container with id 4d881a1c2010ca43d40b3fa03ce715cdf05b6576ebc1b1f270048c484d217408
Sep 11 15:55:05.293384 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.293368    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jf2w"
Sep 11 15:55:05.298267 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.298149    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bdmc8"
Sep 11 15:55:05.303930 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.303906    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025941d9_b594_46d9_b5b6_e212db101199.slice/crio-a0052cea8ba440c3c51db6bf969e9786eb227f81a3aaebd08ad86f1ee5c4738f WatchSource:0}: Error finding container a0052cea8ba440c3c51db6bf969e9786eb227f81a3aaebd08ad86f1ee5c4738f: Status 404 returned error can't find the container with id a0052cea8ba440c3c51db6bf969e9786eb227f81a3aaebd08ad86f1ee5c4738f
Sep 11 15:55:05.306528 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.306507    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430c3e42_d8d6_4937_ab4a_65c6dc9ae962.slice/crio-30f070141558be018f4ffd899b3851c50d9cc773f76c6b668c2dc3c1501b4995 WatchSource:0}: Error finding container 30f070141558be018f4ffd899b3851c50d9cc773f76c6b668c2dc3c1501b4995: Status 404 returned error can't find the container with id 30f070141558be018f4ffd899b3851c50d9cc773f76c6b668c2dc3c1501b4995
Sep 11 15:55:05.321879 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.321834    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cjzgs"
Sep 11 15:55:05.331368 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.331187    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ad41d4_ca46_4cdd_98b8_e198ba5723e3.slice/crio-c5f5693092b9ff14fde6d0bc6426f64b657863f3883c259d4a416618ce2f8985 WatchSource:0}: Error finding container c5f5693092b9ff14fde6d0bc6426f64b657863f3883c259d4a416618ce2f8985: Status 404 returned error can't find the container with id c5f5693092b9ff14fde6d0bc6426f64b657863f3883c259d4a416618ce2f8985
Sep 11 15:55:05.335403 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.335383    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:05.343621 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.343598    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9178ca_3477_4a26_b317_af72dd06fb8f.slice/crio-ceb6b3e9d57b27c7b833a574c2591d1692d4096a01a0d27de523112c16a8fcab WatchSource:0}: Error finding container ceb6b3e9d57b27c7b833a574c2591d1692d4096a01a0d27de523112c16a8fcab: Status 404 returned error can't find the container with id ceb6b3e9d57b27c7b833a574c2591d1692d4096a01a0d27de523112c16a8fcab
Sep 11 15:55:05.344337 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.344318    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-78ppt"
Sep 11 15:55:05.352738 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.352715    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabccca7_3201_4e1e_84d7_ef82f064128a.slice/crio-f7df60c5349131c7dd8b062aa0570be987cbc4b74bb29de6ce6cb26f7208ff68 WatchSource:0}: Error finding container f7df60c5349131c7dd8b062aa0570be987cbc4b74bb29de6ce6cb26f7208ff68: Status 404 returned error can't find the container with id f7df60c5349131c7dd8b062aa0570be987cbc4b74bb29de6ce6cb26f7208ff68
Sep 11 15:55:05.365393 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.365370    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lhrj7"
Sep 11 15:55:05.374725 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:05.374701    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cab6e56_d6b3_4161_b5ca_34d418afb13f.slice/crio-26ba2e098529816dfad39ed779fd597249b952e2f9d8a2d59a5e3dbd44004bb1 WatchSource:0}: Error finding container 26ba2e098529816dfad39ed779fd597249b952e2f9d8a2d59a5e3dbd44004bb1: Status 404 returned error can't find the container with id 26ba2e098529816dfad39ed779fd597249b952e2f9d8a2d59a5e3dbd44004bb1
Sep 11 15:55:05.648095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.647482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:05.648095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.647528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:05.648095 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.647638    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:05.648095 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.647653    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:05.648095 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.647714    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs podName:e4a20019-f1cd-4ee9-bd36-ba1d1e50a527 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:06.647693971 +0000 UTC m=+3.071511649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs") pod "network-metrics-daemon-hspt8" (UID: "e4a20019-f1cd-4ee9-bd36-ba1d1e50a527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:05.648095 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.647744    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret podName:c81f8ab6-37ce-477f-9eae-90441a29d9b3 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:06.647733868 +0000 UTC m=+3.071551545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret") pod "global-pull-secret-syncer-nnlqc" (UID: "c81f8ab6-37ce-477f-9eae-90441a29d9b3") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:05.749113 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:05.748296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:05.749113 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.748466    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:05.749113 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.748483    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:05.749113 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.748497    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5v5wh for pod openshift-network-diagnostics/network-check-target-dvfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:05.749113 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:05.748560    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh podName:d683e04d-faba-49bc-be34-f6e201a52d05 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:06.748539233 +0000 UTC m=+3.172356924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5wh" (UniqueName: "kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh") pod "network-check-target-dvfp8" (UID: "d683e04d-faba-49bc-be34-f6e201a52d05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:06.010397 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.010321    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:50:04 +0000 UTC" deadline="2028-05-29 05:35:19.771524788 +0000 UTC"
Sep 11 15:55:06.010397 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.010394    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15013h40m13.761135377s"
Sep 11 15:55:06.070675 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.070607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"67c01a9ad6d35b61be659f6d32e178b5","Type":"ContainerStarted","Data":"1f0d864eeab0d544a6e0ed274634d6730e446bba056fcb56e721a40889dadb21"}
Sep 11 15:55:06.087330 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.087279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lhrj7" event={"ID":"4cab6e56-d6b3-4161-b5ca-34d418afb13f","Type":"ContainerStarted","Data":"26ba2e098529816dfad39ed779fd597249b952e2f9d8a2d59a5e3dbd44004bb1"}
Sep 11 15:55:06.097906 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.097657    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-78ppt" event={"ID":"dabccca7-3201-4e1e-84d7-ef82f064128a","Type":"ContainerStarted","Data":"f7df60c5349131c7dd8b062aa0570be987cbc4b74bb29de6ce6cb26f7208ff68"}
Sep 11 15:55:06.123532 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.123491    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cjzgs" event={"ID":"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3","Type":"ContainerStarted","Data":"c5f5693092b9ff14fde6d0bc6426f64b657863f3883c259d4a416618ce2f8985"}
Sep 11 15:55:06.129498 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.129462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bdmc8" event={"ID":"430c3e42-d8d6-4937-ab4a-65c6dc9ae962","Type":"ContainerStarted","Data":"30f070141558be018f4ffd899b3851c50d9cc773f76c6b668c2dc3c1501b4995"}
Sep 11 15:55:06.161722 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.161682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv" event={"ID":"bd2c946e-365c-4f0f-88e3-652083f474cb","Type":"ContainerStarted","Data":"93fc898a4b38ef296b029836be0d27b1bc45b47a5a3544c79c5f5b8f5ef18c6f"}
Sep 11 15:55:06.176084 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.176032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wl7tc" event={"ID":"5a5b87e4-2e32-4545-947d-4482ab103ac5","Type":"ContainerStarted","Data":"3cb072cc69a148921df15b9f3966caadb414fb73545a7231deb2f37737d9ac8d"}
Sep 11 15:55:06.201513 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.201470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"fa926f4c1f4ea217ab8dc7d0d74214a4","Type":"ContainerStarted","Data":"35f4f7abbc65cb605a43e6916021026dcb87b13d829a1edf43bef96b3fad314e"}
Sep 11 15:55:06.212554 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.212513    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:06.212554 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.212520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"ceb6b3e9d57b27c7b833a574c2591d1692d4096a01a0d27de523112c16a8fcab"}
Sep 11 15:55:06.224337 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.224222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerStarted","Data":"a0052cea8ba440c3c51db6bf969e9786eb227f81a3aaebd08ad86f1ee5c4738f"}
Sep 11 15:55:06.236957 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.236918    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k6sqh" event={"ID":"3412bf23-2b1c-47a3-8cfe-89e18522785a","Type":"ContainerStarted","Data":"4d881a1c2010ca43d40b3fa03ce715cdf05b6576ebc1b1f270048c484d217408"}
Sep 11 15:55:06.295544 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.295458    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:06.656022 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.655848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:06.656022 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.655903    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:06.656287 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.656053    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:06.656287 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.656118    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret podName:c81f8ab6-37ce-477f-9eae-90441a29d9b3 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:08.656099967 +0000 UTC m=+5.079917642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret") pod "global-pull-secret-syncer-nnlqc" (UID: "c81f8ab6-37ce-477f-9eae-90441a29d9b3") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:06.656414 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.656346    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:06.656414 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.656407    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs podName:e4a20019-f1cd-4ee9-bd36-ba1d1e50a527 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:08.656391041 +0000 UTC m=+5.080208718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs") pod "network-metrics-daemon-hspt8" (UID: "e4a20019-f1cd-4ee9-bd36-ba1d1e50a527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:06.736031 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.736000    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:06.757640 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:06.757603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:06.757871 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.757804    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:06.757871 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.757826    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:06.757871 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.757839    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5v5wh for pod openshift-network-diagnostics/network-check-target-dvfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:06.758037 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:06.757897    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh podName:d683e04d-faba-49bc-be34-f6e201a52d05 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:08.757879108 +0000 UTC m=+5.181696786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5wh" (UniqueName: "kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh") pod "network-check-target-dvfp8" (UID: "d683e04d-faba-49bc-be34-f6e201a52d05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:07.011291 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:07.011200    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:50:04 +0000 UTC" deadline="2028-04-17 19:30:01.160078958 +0000 UTC"
Sep 11 15:55:07.011291 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:07.011251    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14019h34m54.148832676s"
Sep 11 15:55:07.028942 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:07.028160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:07.028942 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:07.028281    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:07.028942 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:07.028679    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:07.028942 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:07.028772    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:07.028942 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:07.028825    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:07.028942 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:07.028898    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:08.673404 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:08.673366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:08.673938 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:08.673419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:08.673938 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.673583    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:08.673938 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.673652    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret podName:c81f8ab6-37ce-477f-9eae-90441a29d9b3 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:12.673632128 +0000 UTC m=+9.097449806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret") pod "global-pull-secret-syncer-nnlqc" (UID: "c81f8ab6-37ce-477f-9eae-90441a29d9b3") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:08.674114 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.674026    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:08.674114 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.674074    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs podName:e4a20019-f1cd-4ee9-bd36-ba1d1e50a527 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:12.674058636 +0000 UTC m=+9.097876313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs") pod "network-metrics-daemon-hspt8" (UID: "e4a20019-f1cd-4ee9-bd36-ba1d1e50a527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:08.774694 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:08.774657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:08.774878 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.774831    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:08.774878 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.774854    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:08.774878 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.774867    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5v5wh for pod openshift-network-diagnostics/network-check-target-dvfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:08.775031 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:08.774935    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh podName:d683e04d-faba-49bc-be34-f6e201a52d05 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:12.774915365 +0000 UTC m=+9.198733060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5wh" (UniqueName: "kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh") pod "network-check-target-dvfp8" (UID: "d683e04d-faba-49bc-be34-f6e201a52d05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:09.028299 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:09.028259    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:09.028489 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:09.028408    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:09.028593 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:09.028574    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:09.028684 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:09.028665    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:09.028733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:09.028696    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:09.028810 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:09.028789    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:11.028036 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:11.027996    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:11.028036 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:11.028031    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:11.028539 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:11.028132    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:11.028539 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:11.028142    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:11.028539 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:11.028263    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:11.028539 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:11.028359    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:12.708417 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:12.708376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:12.708997 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:12.708432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:12.708997 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.708543    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:12.708997 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.708563    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:12.708997 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.708620    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs podName:e4a20019-f1cd-4ee9-bd36-ba1d1e50a527 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:20.708602516 +0000 UTC m=+17.132420188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs") pod "network-metrics-daemon-hspt8" (UID: "e4a20019-f1cd-4ee9-bd36-ba1d1e50a527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:12.708997 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.708644    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret podName:c81f8ab6-37ce-477f-9eae-90441a29d9b3 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:20.708637092 +0000 UTC m=+17.132454764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret") pod "global-pull-secret-syncer-nnlqc" (UID: "c81f8ab6-37ce-477f-9eae-90441a29d9b3") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:12.809567 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:12.809529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:12.809760 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.809683    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:12.809760 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.809715    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:12.809760 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.809729    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5v5wh for pod openshift-network-diagnostics/network-check-target-dvfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:12.809924 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:12.809798    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh podName:d683e04d-faba-49bc-be34-f6e201a52d05 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:20.809779199 +0000 UTC m=+17.233596885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5wh" (UniqueName: "kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh") pod "network-check-target-dvfp8" (UID: "d683e04d-faba-49bc-be34-f6e201a52d05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:13.028088 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:13.028054    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:13.028311 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:13.028177    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:13.028562 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:13.028542    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:13.028695 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:13.028649    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:13.028757 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:13.028736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:13.028843 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:13.028821    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:15.028541 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:15.028494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:15.028541 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:15.028523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:15.029056 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:15.028514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:15.029056 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:15.028647    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:15.029056 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:15.028733    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:15.029056 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:15.028824    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:17.028674 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:17.028598    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:17.029093 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:17.028593    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:17.029093 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:17.028735    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:17.029093 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:17.028834    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:17.029093 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:17.028607    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:17.029093 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:17.028947    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:19.028764 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:19.028597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:19.028764 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:19.028646    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:19.028764 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:19.028597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:19.028764 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:19.028745    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:19.029369 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:19.028806    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:19.029369 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:19.028908    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:20.766744 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:20.766702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:20.766744 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:20.766757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:20.767265 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.766840    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:20.767265 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.766843    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:20.767265 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.766908    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret podName:c81f8ab6-37ce-477f-9eae-90441a29d9b3 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:36.766888832 +0000 UTC m=+33.190706514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret") pod "global-pull-secret-syncer-nnlqc" (UID: "c81f8ab6-37ce-477f-9eae-90441a29d9b3") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:20.767265 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.766947    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs podName:e4a20019-f1cd-4ee9-bd36-ba1d1e50a527 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:36.766930658 +0000 UTC m=+33.190748334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs") pod "network-metrics-daemon-hspt8" (UID: "e4a20019-f1cd-4ee9-bd36-ba1d1e50a527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:20.867894 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:20.867854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:20.868095 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.868063    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:20.868095 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.868094    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:20.868305 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.868133    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5v5wh for pod openshift-network-diagnostics/network-check-target-dvfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:20.868305 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:20.868195    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh podName:d683e04d-faba-49bc-be34-f6e201a52d05 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:36.868179728 +0000 UTC m=+33.291997413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5wh" (UniqueName: "kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh") pod "network-check-target-dvfp8" (UID: "d683e04d-faba-49bc-be34-f6e201a52d05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:21.028416 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:21.028332    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:21.028580 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:21.028344    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:21.028580 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:21.028471    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:21.028580 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:21.028343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:21.028580 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:21.028556    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:21.028752 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:21.028616    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:23.028773 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:23.028740    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:23.028773 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:23.028770    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:23.029446 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:23.028741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:23.029446 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:23.028891    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:23.029446 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:23.029011    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:23.029446 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:23.029084    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:25.029384 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.027941    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:25.029384 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:25.028480    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:25.029384 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.029007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:25.029384 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:25.029127    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:25.029384 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.029203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:25.029384 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:25.029318    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:25.295789 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.295753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"fa926f4c1f4ea217ab8dc7d0d74214a4","Type":"ContainerStarted","Data":"cc6ca3c26d09db1fe75971778bee8678ac1f76a3b6705e946e2ba06de56f51ec"}
Sep 11 15:55:25.298788 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.298757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"ab89c3ccd6ef27450b2681fd47388a39ffbddffa84035196cf364cce0cee8d08"}
Sep 11 15:55:25.298902 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.298831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"d0920ca465be354e8445adf9e517441ff8b5716e5a29dad30c143b9f37532473"}
Sep 11 15:55:25.298902 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.298875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"dee8822cf0ca53de7dfc13d170417df0c03d12b6f570efe5a644098ffbf3debd"}
Sep 11 15:55:25.298978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.298915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"ba11fefa9ab14959ea8bcd509ab8e726c4d16f9f92bc4d96d4dc6832b23f9201"}
Sep 11 15:55:25.298978 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.298960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"badc942a0f4997ee450ef64630057412ba94a553b189f9849206a3422bf609bb"}
Sep 11 15:55:25.300387 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.300359    2559 generic.go:356] "Generic (PLEG): container finished" podID="025941d9-b594-46d9-b5b6-e212db101199" containerID="44518d633f825c65e281565c7ee2c5bf91a8be6c6be4b22a7c99fe3d8d98c434" exitCode=0
Sep 11 15:55:25.300510 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.300436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerDied","Data":"44518d633f825c65e281565c7ee2c5bf91a8be6c6be4b22a7c99fe3d8d98c434"}
Sep 11 15:55:25.301940 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.301912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k6sqh" event={"ID":"3412bf23-2b1c-47a3-8cfe-89e18522785a","Type":"ContainerStarted","Data":"f11b8341eb004414e447a86ccd1e4f36908ed4d0d276107fe7896c421e08ab89"}
Sep 11 15:55:25.303219 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.303194    2559 generic.go:356] "Generic (PLEG): container finished" podID="67c01a9ad6d35b61be659f6d32e178b5" containerID="4bd938b14e8b3175986e1fe6a649bf7bb32c7f178e6c38c6e39b908e9992c4fc" exitCode=0
Sep 11 15:55:25.303330 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.303277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"67c01a9ad6d35b61be659f6d32e178b5","Type":"ContainerDied","Data":"4bd938b14e8b3175986e1fe6a649bf7bb32c7f178e6c38c6e39b908e9992c4fc"}
Sep 11 15:55:25.304698 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.304675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lhrj7" event={"ID":"4cab6e56-d6b3-4161-b5ca-34d418afb13f","Type":"ContainerStarted","Data":"4bf5275078dc0c7095c8bf0ec62af5521101edac33f03812893b2ecb46efedaf"}
Sep 11 15:55:25.306112 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.306091    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-78ppt" event={"ID":"dabccca7-3201-4e1e-84d7-ef82f064128a","Type":"ContainerStarted","Data":"faba214d1c39451e4463a93a6dce8c6ff7403ef8e1ce78ec81495870beb9b86f"}
Sep 11 15:55:25.307491 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.307472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bdmc8" event={"ID":"430c3e42-d8d6-4937-ab4a-65c6dc9ae962","Type":"ContainerStarted","Data":"8f0eb31e37787126b7bb9c7f37287bca11cfee0346c0c01ccd13e7805b686b96"}
Sep 11 15:55:25.308922 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.308898    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv" event={"ID":"bd2c946e-365c-4f0f-88e3-652083f474cb","Type":"ContainerStarted","Data":"6542dcf991a24a05d44078c8297e1e06d89f068b4dd805441e1f302f483f84b2"}
Sep 11 15:55:25.310033 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.309993    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" podStartSLOduration=21.309978839 podStartE2EDuration="21.309978839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:25.309212458 +0000 UTC m=+21.733030152" watchObservedRunningTime="2026-09-11 15:55:25.309978839 +0000 UTC m=+21.733796532"
Sep 11 15:55:25.310357 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.310337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wl7tc" event={"ID":"5a5b87e4-2e32-4545-947d-4482ab103ac5","Type":"ContainerStarted","Data":"39db4a35ca3c581101782ee79dda285276c2025aab450b0eaf2b938ddbcf8ecb"}
Sep 11 15:55:25.323526 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.323479    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-78ppt" podStartSLOduration=2.362426621 podStartE2EDuration="21.323457673s" totalImagesPullingTime="18.961031052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:25.323027952 +0000 UTC m=+21.746845667" watchObservedRunningTime="2026-09-11 15:55:25.323457673 +0000 UTC m=+21.747275369"
Sep 11 15:55:25.337733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.337645    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lhrj7" podStartSLOduration=2.399006666 podStartE2EDuration="21.337630513s" totalImagesPullingTime="18.938623847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:25.337320756 +0000 UTC m=+21.761138463" watchObservedRunningTime="2026-09-11 15:55:25.337630513 +0000 UTC m=+21.761448208"
Sep 11 15:55:25.356082 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.356026    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k6sqh" podStartSLOduration=2.330382111 podStartE2EDuration="21.356009909s" totalImagesPullingTime="19.025627798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:25.355421827 +0000 UTC m=+21.779239544" watchObservedRunningTime="2026-09-11 15:55:25.356009909 +0000 UTC m=+21.779827603"
Sep 11 15:55:25.412864 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.412814    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bdmc8" podStartSLOduration=2.12480046 podStartE2EDuration="21.412799174s" totalImagesPullingTime="19.287998714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:25.412455813 +0000 UTC m=+21.836273527" watchObservedRunningTime="2026-09-11 15:55:25.412799174 +0000 UTC m=+21.836616868"
Sep 11 15:55:25.429117 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:25.429068    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wl7tc" podStartSLOduration=2.387590546 podStartE2EDuration="21.429052725s" totalImagesPullingTime="19.041462179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:25.428693817 +0000 UTC m=+21.852511530" watchObservedRunningTime="2026-09-11 15:55:25.429052725 +0000 UTC m=+21.852870419"
Sep 11 15:55:26.315438 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:26.315407    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"38c7118ee4786cd8be19dc9cb4df1c214f57f3d17bbfe993c00f014fdd41ebd1"}
Sep 11 15:55:26.317714 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:26.317663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"67c01a9ad6d35b61be659f6d32e178b5","Type":"ContainerStarted","Data":"d6db7afc11c7fa2b13f5df6f02c57749dcb1db49792ed1079f844790334e9243"}
Sep 11 15:55:26.319274 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:26.319245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cjzgs" event={"ID":"b1ad41d4-ca46-4cdd-98b8-e198ba5723e3","Type":"ContainerStarted","Data":"a812b37f87a4ae9780b18dc5053e4528a9e548b2fd2fb5fe2e06f8f55d431678"}
Sep 11 15:55:26.333446 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:26.333388    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" podStartSLOduration=22.333366762 podStartE2EDuration="22.333366762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:26.333317751 +0000 UTC m=+22.757135469" watchObservedRunningTime="2026-09-11 15:55:26.333366762 +0000 UTC m=+22.757184457"
Sep 11 15:55:26.349314 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:26.349226    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cjzgs" podStartSLOduration=3.366641896 podStartE2EDuration="22.349205442s" totalImagesPullingTime="18.982563546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:26.348674925 +0000 UTC m=+22.772492639" watchObservedRunningTime="2026-09-11 15:55:26.349205442 +0000 UTC m=+22.773023136"
Sep 11 15:55:26.576307 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:26.575926    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 15:55:27.004997 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.004878    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T15:55:26.575944528Z","UUID":"e8e68d72-93e5-4f78-8ab7-5fc7491f7ff0","Handler":null,"Name":"","Endpoint":""}
Sep 11 15:55:27.006844 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.006817    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 15:55:27.006996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.006862    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 15:55:27.028186 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.028153    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:27.028374 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.028192    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:27.028374 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.028191    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:27.028374 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:27.028297    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:27.028374 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:27.028364    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:27.028633 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:27.028436    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:27.097127 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.097086    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:27.097977 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.097953    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:27.325194 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.325101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv" event={"ID":"bd2c946e-365c-4f0f-88e3-652083f474cb","Type":"ContainerStarted","Data":"0980537664bee3041ba53c9b26d2d5cceee4b93f74c4a4e7a2b1bbc261e04749"}
Sep 11 15:55:27.325676 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.325412    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:27.326115 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:27.326053    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wl7tc"
Sep 11 15:55:28.330071 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:28.329794    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv" event={"ID":"bd2c946e-365c-4f0f-88e3-652083f474cb","Type":"ContainerStarted","Data":"8add65fcc3bd067eb4208d745585231adc69ddba10457b87e4ca88d7747733c9"}
Sep 11 15:55:28.333185 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:28.333150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"1bbf67517f9f59980da417814423eca544ddf01a08bb7c0b4dc924e29ed7a4e3"}
Sep 11 15:55:28.351720 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:28.351665    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9jtv" podStartSLOduration=2.731111271 podStartE2EDuration="24.351647741s" totalImagesPullingTime="21.62053647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:28.351216162 +0000 UTC m=+24.775033875" watchObservedRunningTime="2026-09-11 15:55:28.351647741 +0000 UTC m=+24.775465435"
Sep 11 15:55:29.027894 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:29.027857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:29.028090 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:29.027912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:29.028090 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:29.027911    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:29.028090 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:29.027973    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:29.028298 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:29.028110    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:29.028298 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:29.028195    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:30.342396 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:30.342121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" event={"ID":"9e9178ca-3477-4a26-b317-af72dd06fb8f","Type":"ContainerStarted","Data":"b899755e90058863fc2b94d51b6da102faa539324cf102b75a7a8e673f156d77"}
Sep 11 15:55:30.342834 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:30.342449    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:30.342834 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:30.342503    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:30.358617 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:30.358492    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:30.401815 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:30.401760    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn" podStartSLOduration=7.093128077 podStartE2EDuration="26.401744297s" totalImagesPullingTime="19.30861622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:30.373604749 +0000 UTC m=+26.797422448" watchObservedRunningTime="2026-09-11 15:55:30.401744297 +0000 UTC m=+26.825561991"
Sep 11 15:55:31.028464 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:31.028422    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:31.028673 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:31.028541    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:31.028673 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:31.028557    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:31.028673 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:31.028572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:31.028673 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:31.028635    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:31.028829 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:31.028699    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:31.348812 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:31.348717    2559 generic.go:356] "Generic (PLEG): container finished" podID="025941d9-b594-46d9-b5b6-e212db101199" containerID="811df4d2633f4681f5d39116e555f653e2edf6051a1594a55a24dbc3ac41ffa4" exitCode=0
Sep 11 15:55:31.349275 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:31.348820    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerDied","Data":"811df4d2633f4681f5d39116e555f653e2edf6051a1594a55a24dbc3ac41ffa4"}
Sep 11 15:55:31.349351 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:31.349323    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:31.365106 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:31.365078    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:55:32.267930 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:32.267782    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hspt8"]
Sep 11 15:55:32.267930 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:32.267908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:32.268142 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:32.268038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:32.271277 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:32.271224    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nnlqc"]
Sep 11 15:55:32.271420 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:32.271339    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:32.271478 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:32.271446    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:32.271772 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:32.271747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvfp8"]
Sep 11 15:55:32.271879 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:32.271828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:32.271927 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:32.271898    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:33.355618 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:33.355425    2559 generic.go:356] "Generic (PLEG): container finished" podID="025941d9-b594-46d9-b5b6-e212db101199" containerID="f198115aca5a37098d6b15b5ed0699709bd780a033e7aa9a9afafc3d04c64f42" exitCode=0
Sep 11 15:55:33.356533 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:33.355513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerDied","Data":"f198115aca5a37098d6b15b5ed0699709bd780a033e7aa9a9afafc3d04c64f42"}
Sep 11 15:55:34.028541 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:34.028511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:34.028716 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:34.028623    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:34.028716 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:34.028631    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:34.028716 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:34.028694    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:34.028879 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:34.028742    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:34.028879 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:34.028805    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:34.360719 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:34.360687    2559 generic.go:356] "Generic (PLEG): container finished" podID="025941d9-b594-46d9-b5b6-e212db101199" containerID="c6448f823418ac78f50a969c22397a2af63b1b85090e723b0bcef92bad00b84a" exitCode=0
Sep 11 15:55:34.361064 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:34.360734    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerDied","Data":"c6448f823418ac78f50a969c22397a2af63b1b85090e723b0bcef92bad00b84a"}
Sep 11 15:55:36.028635 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:36.028587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:36.029083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:36.028587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:36.029083 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.028715    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnlqc" podUID="c81f8ab6-37ce-477f-9eae-90441a29d9b3"
Sep 11 15:55:36.029083 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.028807    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hspt8" podUID="e4a20019-f1cd-4ee9-bd36-ba1d1e50a527"
Sep 11 15:55:36.029083 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:36.028600    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:36.029083 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.028889    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvfp8" podUID="d683e04d-faba-49bc-be34-f6e201a52d05"
Sep 11 15:55:36.783500 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:36.783466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:36.783500 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:36.783513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:36.783758 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.783639    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:36.783758 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.783662    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:36.783758 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.783714    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs podName:e4a20019-f1cd-4ee9-bd36-ba1d1e50a527 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:08.783694044 +0000 UTC m=+65.207511739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs") pod "network-metrics-daemon-hspt8" (UID: "e4a20019-f1cd-4ee9-bd36-ba1d1e50a527") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:36.783758 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.783733    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret podName:c81f8ab6-37ce-477f-9eae-90441a29d9b3 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:08.783725798 +0000 UTC m=+65.207543470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret") pod "global-pull-secret-syncer-nnlqc" (UID: "c81f8ab6-37ce-477f-9eae-90441a29d9b3") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:36.884595 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:36.884555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:36.884797 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.884743    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:36.884797 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.884773    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:36.884797 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.884787    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5v5wh for pod openshift-network-diagnostics/network-check-target-dvfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:36.884958 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:36.884857    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh podName:d683e04d-faba-49bc-be34-f6e201a52d05 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:08.884837394 +0000 UTC m=+65.308655080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5wh" (UniqueName: "kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh") pod "network-check-target-dvfp8" (UID: "d683e04d-faba-49bc-be34-f6e201a52d05") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:37.380958 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.380924    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeReady"
Sep 11 15:55:37.381471 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.381080    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 15:55:37.425058 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.424983    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fc6cb946f-j94ns"]
Sep 11 15:55:37.452297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.452264    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jtzf7"]
Sep 11 15:55:37.452491 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.452357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.454977 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.454949    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 15:55:37.455144 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.455088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6wjl\""
Sep 11 15:55:37.455231 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.455209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 15:55:37.455559 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.455539    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 15:55:37.466987 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.466119    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc6cb946f-j94ns"]
Sep 11 15:55:37.466987 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.466174    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zl6nf"]
Sep 11 15:55:37.467188 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.467130    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.470044 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.470021    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 15:55:37.470228 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.470200    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 15:55:37.470699 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.470681    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 15:55:37.470803 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.470698    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvv4c\""
Sep 11 15:55:37.488221 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.488189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jtzf7"]
Sep 11 15:55:37.488377 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.488274    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zl6nf"]
Sep 11 15:55:37.488377 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.488201    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.491114 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.491088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dxw2k\""
Sep 11 15:55:37.491599 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.491579    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 15:55:37.491720 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.491579    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 15:55:37.491720 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.491623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 15:55:37.503635 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.503610    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jpsls"]
Sep 11 15:55:37.523386 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.523351    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jpsls"]
Sep 11 15:55:37.523566 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.523430    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.526447 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.526408    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4ddf\""
Sep 11 15:55:37.526447 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.526447    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 15:55:37.526633 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.526517    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 15:55:37.526633 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.526408    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 15:55:37.526842 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.526730    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 15:55:37.588873 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.588831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-image-registry-private-configuration\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.588873 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.588876    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.589130 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.588902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59b28b96-2421-41b3-8d69-842d2e47799b-data-volume\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.589130 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.588929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-ca-trust-extracted\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.589130 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.588954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-trusted-ca\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.589130 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.588975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-bound-sa-token\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.589130 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.588989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/852587b1-0a81-4241-a4bc-af2730e72a53-config-volume\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.589130 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/852587b1-0a81-4241-a4bc-af2730e72a53-tmp-dir\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.589130 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af-cert\") pod \"ingress-canary-zl6nf\" (UID: \"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af\") " pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589142    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-installation-pull-secrets\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589177    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb98c\" (UniqueName: \"kubernetes.io/projected/59b28b96-2421-41b3-8d69-842d2e47799b-kube-api-access-vb98c\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-certificates\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4sf\" (UniqueName: \"kubernetes.io/projected/852587b1-0a81-4241-a4bc-af2730e72a53-kube-api-access-cr4sf\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5bb6\" (UniqueName: \"kubernetes.io/projected/1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af-kube-api-access-v5bb6\") pod \"ingress-canary-zl6nf\" (UID: \"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af\") " pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2vv\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-kube-api-access-sm2vv\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59b28b96-2421-41b3-8d69-842d2e47799b-crio-socket\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.589507 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.589446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.690857 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.690770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2vv\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-kube-api-access-sm2vv\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.690857 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.690816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59b28b96-2421-41b3-8d69-842d2e47799b-crio-socket\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.691078 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.690946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.691078 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.690995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-image-registry-private-configuration\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691078 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691078 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59b28b96-2421-41b3-8d69-842d2e47799b-crio-socket\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.691078 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59b28b96-2421-41b3-8d69-842d2e47799b-data-volume\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.691078 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.691072    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.691161    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.691179    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc6cb946f-j94ns: secret "image-registry-tls" not found
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-ca-trust-extracted\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.691277    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls podName:85831938-67dc-49ff-9d0f-9b62a3a6dc0d nodeName:}" failed. No retries permitted until 2026-09-11 15:55:38.191251609 +0000 UTC m=+34.615069295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls") pod "image-registry-7fc6cb946f-j94ns" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d") : secret "image-registry-tls" not found
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.691312    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:38.191301222 +0000 UTC m=+34.615118895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : configmap "kube-rbac-proxy" not found
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-trusted-ca\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-bound-sa-token\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691388 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/852587b1-0a81-4241-a4bc-af2730e72a53-config-volume\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/852587b1-0a81-4241-a4bc-af2730e72a53-tmp-dir\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af-cert\") pod \"ingress-canary-zl6nf\" (UID: \"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af\") " pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-installation-pull-secrets\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb98c\" (UniqueName: \"kubernetes.io/projected/59b28b96-2421-41b3-8d69-842d2e47799b-kube-api-access-vb98c\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691599    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-certificates\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691607    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-ca-trust-extracted\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.691728 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691451    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59b28b96-2421-41b3-8d69-842d2e47799b-data-volume\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.692146 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.691916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/852587b1-0a81-4241-a4bc-af2730e72a53-tmp-dir\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.692146 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.692038    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:37.692146 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.692060    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/852587b1-0a81-4241-a4bc-af2730e72a53-config-volume\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.692146 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.692124    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:38.192096253 +0000 UTC m=+34.615913941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:37.692332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.692165    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.692332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.692203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4sf\" (UniqueName: \"kubernetes.io/projected/852587b1-0a81-4241-a4bc-af2730e72a53-kube-api-access-cr4sf\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.692332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.692251    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5bb6\" (UniqueName: \"kubernetes.io/projected/1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af-kube-api-access-v5bb6\") pod \"ingress-canary-zl6nf\" (UID: \"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af\") " pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.692496 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.692479    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:37.692554 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:37.692541    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls podName:852587b1-0a81-4241-a4bc-af2730e72a53 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:38.192526008 +0000 UTC m=+34.616343700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls") pod "dns-default-jtzf7" (UID: "852587b1-0a81-4241-a4bc-af2730e72a53") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:37.692646 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.692597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-certificates\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.693190 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.693166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-trusted-ca\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.696273 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.696252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-installation-pull-secrets\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.696273 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.696252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af-cert\") pod \"ingress-canary-zl6nf\" (UID: \"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af\") " pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.696424 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.696251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-image-registry-private-configuration\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.704150 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.704095    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2vv\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-kube-api-access-sm2vv\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.704593 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.704566    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-bound-sa-token\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:37.704748 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.704720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4sf\" (UniqueName: \"kubernetes.io/projected/852587b1-0a81-4241-a4bc-af2730e72a53-kube-api-access-cr4sf\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:37.705759 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.705708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5bb6\" (UniqueName: \"kubernetes.io/projected/1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af-kube-api-access-v5bb6\") pod \"ingress-canary-zl6nf\" (UID: \"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af\") " pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.706082 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.706063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb98c\" (UniqueName: \"kubernetes.io/projected/59b28b96-2421-41b3-8d69-842d2e47799b-kube-api-access-vb98c\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:37.799147 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.799104    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zl6nf"
Sep 11 15:55:37.837158 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.837125    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"]
Sep 11 15:55:37.852630 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.852601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"]
Sep 11 15:55:37.852800 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.852750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"
Sep 11 15:55:37.855536 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.855510    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 15:55:37.855666 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.855546    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 15:55:37.855910 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.855894    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zk57g\""
Sep 11 15:55:37.994926 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:37.994838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lxp\" (UniqueName: \"kubernetes.io/projected/c6e060cf-52e6-4bd5-a296-51ac19706df0-kube-api-access-x4lxp\") pod \"migrator-8b4d697c5-chm5p\" (UID: \"c6e060cf-52e6-4bd5-a296-51ac19706df0\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"
Sep 11 15:55:38.028173 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.028137    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:55:38.028173 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.028171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:55:38.028432 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.028139    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:55:38.031352 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.031197    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zsbjs\""
Sep 11 15:55:38.031352 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.031313    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 15:55:38.031352 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.031329    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 15:55:38.031352 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.031347    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lsspg\""
Sep 11 15:55:38.031764 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.031746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 15:55:38.031869 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.031817    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 15:55:38.096297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.096257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lxp\" (UniqueName: \"kubernetes.io/projected/c6e060cf-52e6-4bd5-a296-51ac19706df0-kube-api-access-x4lxp\") pod \"migrator-8b4d697c5-chm5p\" (UID: \"c6e060cf-52e6-4bd5-a296-51ac19706df0\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"
Sep 11 15:55:38.105751 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.105716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lxp\" (UniqueName: \"kubernetes.io/projected/c6e060cf-52e6-4bd5-a296-51ac19706df0-kube-api-access-x4lxp\") pod \"migrator-8b4d697c5-chm5p\" (UID: \"c6e060cf-52e6-4bd5-a296-51ac19706df0\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"
Sep 11 15:55:38.165244 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.165193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"
Sep 11 15:55:38.197542 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.197321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:38.197703 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.197567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:38.197703 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197454    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 15:55:38.197703 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.197641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:38.197703 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197682    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:39.197656882 +0000 UTC m=+35.621474566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : configmap "kube-rbac-proxy" not found
Sep 11 15:55:38.197924 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.197752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:38.197924 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197785    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:38.197924 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197809    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 15:55:38.197924 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197829    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc6cb946f-j94ns: secret "image-registry-tls" not found
Sep 11 15:55:38.197924 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197835    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:39.197820591 +0000 UTC m=+35.621638262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:38.197924 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197882    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls podName:85831938-67dc-49ff-9d0f-9b62a3a6dc0d nodeName:}" failed. No retries permitted until 2026-09-11 15:55:39.197860159 +0000 UTC m=+35.621677836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls") pod "image-registry-7fc6cb946f-j94ns" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d") : secret "image-registry-tls" not found
Sep 11 15:55:38.197924 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197915    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:38.198184 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:38.197953    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls podName:852587b1-0a81-4241-a4bc-af2730e72a53 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:39.19794128 +0000 UTC m=+35.621758952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls") pod "dns-default-jtzf7" (UID: "852587b1-0a81-4241-a4bc-af2730e72a53") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:38.828747 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:38.828723    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-78ppt_dabccca7-3201-4e1e-84d7-ef82f064128a/dns-node-resolver/0.log"
Sep 11 15:55:39.028392 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.028363    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p"]
Sep 11 15:55:39.031292 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.031274    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zl6nf"]
Sep 11 15:55:39.032054 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:39.032033    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e060cf_52e6_4bd5_a296_51ac19706df0.slice/crio-f02bd6ad6a352f0d9a4c18fe7f8118dbf71ca09e8e99e77d26641e1f7f3c90c9 WatchSource:0}: Error finding container f02bd6ad6a352f0d9a4c18fe7f8118dbf71ca09e8e99e77d26641e1f7f3c90c9: Status 404 returned error can't find the container with id f02bd6ad6a352f0d9a4c18fe7f8118dbf71ca09e8e99e77d26641e1f7f3c90c9
Sep 11 15:55:39.033814 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:39.033791    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa7ffdf_6c14_451d_88c9_c92eb7ebd9af.slice/crio-2f25d948da5fe6a034bf2632025db1b0d5f7fb677697e8d9a29bce882c2aa11a WatchSource:0}: Error finding container 2f25d948da5fe6a034bf2632025db1b0d5f7fb677697e8d9a29bce882c2aa11a: Status 404 returned error can't find the container with id 2f25d948da5fe6a034bf2632025db1b0d5f7fb677697e8d9a29bce882c2aa11a
Sep 11 15:55:39.207114 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.207069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:39.207350 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.207132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:39.207350 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.207157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:39.207350 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.207216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:39.207350 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207228    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:39.207350 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207317    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:39.207350 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207335    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 15:55:39.207350 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207256    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 15:55:39.207720 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207317    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls podName:852587b1-0a81-4241-a4bc-af2730e72a53 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.207300238 +0000 UTC m=+37.631117911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls") pod "dns-default-jtzf7" (UID: "852587b1-0a81-4241-a4bc-af2730e72a53") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:39.207720 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207409    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.207393051 +0000 UTC m=+37.631210738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:39.207720 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207352    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc6cb946f-j94ns: secret "image-registry-tls" not found
Sep 11 15:55:39.207720 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207425    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.207415607 +0000 UTC m=+37.631233284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : configmap "kube-rbac-proxy" not found
Sep 11 15:55:39.207720 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:39.207496    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls podName:85831938-67dc-49ff-9d0f-9b62a3a6dc0d nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.207481598 +0000 UTC m=+37.631299269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls") pod "image-registry-7fc6cb946f-j94ns" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d") : secret "image-registry-tls" not found
Sep 11 15:55:39.374290 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.374251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zl6nf" event={"ID":"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af","Type":"ContainerStarted","Data":"2f25d948da5fe6a034bf2632025db1b0d5f7fb677697e8d9a29bce882c2aa11a"}
Sep 11 15:55:39.376952 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.376924    2559 generic.go:356] "Generic (PLEG): container finished" podID="025941d9-b594-46d9-b5b6-e212db101199" containerID="bb8b65f133f6ee02275456256095a240962035fd966317d5a7d8bfb5141a4dbc" exitCode=0
Sep 11 15:55:39.377079 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.377000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerDied","Data":"bb8b65f133f6ee02275456256095a240962035fd966317d5a7d8bfb5141a4dbc"}
Sep 11 15:55:39.378266 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:39.378227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p" event={"ID":"c6e060cf-52e6-4bd5-a296-51ac19706df0","Type":"ContainerStarted","Data":"f02bd6ad6a352f0d9a4c18fe7f8118dbf71ca09e8e99e77d26641e1f7f3c90c9"}
Sep 11 15:55:40.048084 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.048050    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-p2jgv"]
Sep 11 15:55:40.064753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.064723    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-p2jgv"]
Sep 11 15:55:40.064932 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.064891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.068528 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.068429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-p6vdz\""
Sep 11 15:55:40.068528 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.068427    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 15:55:40.068528 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.068483    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 15:55:40.068851 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.068787    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 15:55:40.069043 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.069009    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 15:55:40.115843 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.115803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrptp\" (UniqueName: \"kubernetes.io/projected/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-kube-api-access-nrptp\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.116030 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.115853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-signing-key\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.116030 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.115893    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-signing-cabundle\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.216446 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.216415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrptp\" (UniqueName: \"kubernetes.io/projected/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-kube-api-access-nrptp\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.216446 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.216456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-signing-key\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.216697 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.216492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-signing-cabundle\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.217537 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.217499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-signing-cabundle\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.220662 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.220609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-signing-key\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.229186 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.229157    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lhrj7_4cab6e56-d6b3-4161-b5ca-34d418afb13f/node-ca/0.log"
Sep 11 15:55:40.230422 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.230393    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrptp\" (UniqueName: \"kubernetes.io/projected/53d29f07-0e7c-4acb-b51c-a63f0e5f79df-kube-api-access-nrptp\") pod \"service-ca-568869fdc5-p2jgv\" (UID: \"53d29f07-0e7c-4acb-b51c-a63f0e5f79df\") " pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.376553 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.376228    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-p2jgv"
Sep 11 15:55:40.385867 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.385828    2559 generic.go:356] "Generic (PLEG): container finished" podID="025941d9-b594-46d9-b5b6-e212db101199" containerID="62021b6515bc708acdcb8521ac36a1eccba48f839262068ce2ddc2898e6f0c83" exitCode=0
Sep 11 15:55:40.386054 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.385900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerDied","Data":"62021b6515bc708acdcb8521ac36a1eccba48f839262068ce2ddc2898e6f0c83"}
Sep 11 15:55:40.862580 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:40.862437    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-p2jgv"]
Sep 11 15:55:41.224651 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:41.224614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:41.224666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:41.224730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:41.224764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224783    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224847    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224851    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224856    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:45.224838338 +0000 UTC m=+41.648656042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224895    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224909    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc6cb946f-j94ns: secret "image-registry-tls" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224919    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls podName:852587b1-0a81-4241-a4bc-af2730e72a53 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:45.224902181 +0000 UTC m=+41.648719862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls") pod "dns-default-jtzf7" (UID: "852587b1-0a81-4241-a4bc-af2730e72a53") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224943    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:45.224933078 +0000 UTC m=+41.648750751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : configmap "kube-rbac-proxy" not found
Sep 11 15:55:41.225209 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:41.224957    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls podName:85831938-67dc-49ff-9d0f-9b62a3a6dc0d nodeName:}" failed. No retries permitted until 2026-09-11 15:55:45.224949592 +0000 UTC m=+41.648767264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls") pod "image-registry-7fc6cb946f-j94ns" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d") : secret "image-registry-tls" not found
Sep 11 15:55:41.392831 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:41.392800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" event={"ID":"025941d9-b594-46d9-b5b6-e212db101199","Type":"ContainerStarted","Data":"a9cbb265acc09681625c51780cab2664f8026e3f4e46488097b3140bfd62f8be"}
Sep 11 15:55:41.421787 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:41.421720    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7jf2w" podStartSLOduration=6.967995332 podStartE2EDuration="37.421695304s" totalImagesPullingTime="30.453699972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:41.420022325 +0000 UTC m=+37.843840039" watchObservedRunningTime="2026-09-11 15:55:41.421695304 +0000 UTC m=+37.845512993"
Sep 11 15:55:41.480209 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:41.480138    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d29f07_0e7c_4acb_b51c_a63f0e5f79df.slice/crio-2960c5250e2a4635ba5eea2240547907debf932c85761073a339561853a3b2a2 WatchSource:0}: Error finding container 2960c5250e2a4635ba5eea2240547907debf932c85761073a339561853a3b2a2: Status 404 returned error can't find the container with id 2960c5250e2a4635ba5eea2240547907debf932c85761073a339561853a3b2a2
Sep 11 15:55:42.398338 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:42.398061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zl6nf" event={"ID":"1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af","Type":"ContainerStarted","Data":"ef2407420341a72f86124957ff0516ee5af8c933c3964142704830a9aa3c671f"}
Sep 11 15:55:42.400677 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:42.400642    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p" event={"ID":"c6e060cf-52e6-4bd5-a296-51ac19706df0","Type":"ContainerStarted","Data":"1d98be4d41199b9dcaee70acd7b3732fc8f860ae61da90a6963a78e9729bb05c"}
Sep 11 15:55:42.400833 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:42.400708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p" event={"ID":"c6e060cf-52e6-4bd5-a296-51ac19706df0","Type":"ContainerStarted","Data":"cdfc623427a5c5640bf04a92250df7e5ee0aa66f5d341d3a7ababb3daa2e30a5"}
Sep 11 15:55:42.402092 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:42.402060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-p2jgv" event={"ID":"53d29f07-0e7c-4acb-b51c-a63f0e5f79df","Type":"ContainerStarted","Data":"2960c5250e2a4635ba5eea2240547907debf932c85761073a339561853a3b2a2"}
Sep 11 15:55:42.420833 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:42.420779    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zl6nf" podStartSLOduration=2.770679796 podStartE2EDuration="5.420756221s" totalImagesPullingTime="2.650076425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:42.419981927 +0000 UTC m=+38.843799622" watchObservedRunningTime="2026-09-11 15:55:42.420756221 +0000 UTC m=+38.844573919"
Sep 11 15:55:42.440346 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:42.440286    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-chm5p" podStartSLOduration=2.979154667 podStartE2EDuration="5.44027064s" totalImagesPullingTime="2.461115973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:42.44009774 +0000 UTC m=+38.863915455" watchObservedRunningTime="2026-09-11 15:55:42.44027064 +0000 UTC m=+38.864088331"
Sep 11 15:55:44.408451 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:44.408411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-p2jgv" event={"ID":"53d29f07-0e7c-4acb-b51c-a63f0e5f79df","Type":"ContainerStarted","Data":"f9031d6ac83a70470a4dfe9ab3a16f71c84845f945baf065e8ff19782434a67d"}
Sep 11 15:55:45.261535 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:45.261495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:45.261733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:45.261573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:45.261733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:45.261610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:45.261733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:45.261674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:45.261733 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:45.261687    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:45.261973 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:45.261774    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:45.261973 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:45.261875    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 15:55:45.261973 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:45.261886    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc6cb946f-j94ns: secret "image-registry-tls" not found
Sep 11 15:55:45.261973 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:45.261780    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls podName:852587b1-0a81-4241-a4bc-af2730e72a53 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:53.261757578 +0000 UTC m=+49.685575265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls") pod "dns-default-jtzf7" (UID: "852587b1-0a81-4241-a4bc-af2730e72a53") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:45.261973 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:45.261934    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls podName:59b28b96-2421-41b3-8d69-842d2e47799b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:53.261916425 +0000 UTC m=+49.685734103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jpsls" (UID: "59b28b96-2421-41b3-8d69-842d2e47799b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:45.261973 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:55:45.261948    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls podName:85831938-67dc-49ff-9d0f-9b62a3a6dc0d nodeName:}" failed. No retries permitted until 2026-09-11 15:55:53.261939486 +0000 UTC m=+49.685757163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls") pod "image-registry-7fc6cb946f-j94ns" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d") : secret "image-registry-tls" not found
Sep 11 15:55:45.262185 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:45.262162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59b28b96-2421-41b3-8d69-842d2e47799b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:53.330118 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.330077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:53.330663 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.330133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:53.330663 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.330178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:53.334048 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.334010    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59b28b96-2421-41b3-8d69-842d2e47799b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpsls\" (UID: \"59b28b96-2421-41b3-8d69-842d2e47799b\") " pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:53.334160 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.334027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/852587b1-0a81-4241-a4bc-af2730e72a53-metrics-tls\") pod \"dns-default-jtzf7\" (UID: \"852587b1-0a81-4241-a4bc-af2730e72a53\") " pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:53.334160 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.334076    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"image-registry-7fc6cb946f-j94ns\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") " pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:53.368631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.368595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:53.377797 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.377762    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:53.434602 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.434488    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jpsls"
Sep 11 15:55:53.669423 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.669287    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-p2jgv" podStartSLOduration=11.309915112 podStartE2EDuration="13.669264093s" totalImagesPullingTime="2.359348981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:44.443889873 +0000 UTC m=+40.867707588" watchObservedRunningTime="2026-09-11 15:55:53.669264093 +0000 UTC m=+50.093081780"
Sep 11 15:55:53.669705 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.669688    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc6cb946f-j94ns"]
Sep 11 15:55:53.672249 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:53.672197    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85831938_67dc_49ff_9d0f_9b62a3a6dc0d.slice/crio-465dc580c74798cb63e81a7ac0f9262b9d6b5f38983496205b89824415bf3a9d WatchSource:0}: Error finding container 465dc580c74798cb63e81a7ac0f9262b9d6b5f38983496205b89824415bf3a9d: Status 404 returned error can't find the container with id 465dc580c74798cb63e81a7ac0f9262b9d6b5f38983496205b89824415bf3a9d
Sep 11 15:55:53.684132 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.684049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jtzf7"]
Sep 11 15:55:53.686516 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:53.686488    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852587b1_0a81_4241_a4bc_af2730e72a53.slice/crio-c12730ed6b384dc3e361bab652f570f3e7910408f2de3426190d49033eff17d6 WatchSource:0}: Error finding container c12730ed6b384dc3e361bab652f570f3e7910408f2de3426190d49033eff17d6: Status 404 returned error can't find the container with id c12730ed6b384dc3e361bab652f570f3e7910408f2de3426190d49033eff17d6
Sep 11 15:55:53.733213 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:53.733185    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jpsls"]
Sep 11 15:55:53.736376 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:55:53.736346    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b28b96_2421_41b3_8d69_842d2e47799b.slice/crio-2e91891fd97ff43e99045ebc181703b9ab9046802e1f3cddcc3c5e6546c6c2ba WatchSource:0}: Error finding container 2e91891fd97ff43e99045ebc181703b9ab9046802e1f3cddcc3c5e6546c6c2ba: Status 404 returned error can't find the container with id 2e91891fd97ff43e99045ebc181703b9ab9046802e1f3cddcc3c5e6546c6c2ba
Sep 11 15:55:54.438701 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:54.438659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns" event={"ID":"85831938-67dc-49ff-9d0f-9b62a3a6dc0d","Type":"ContainerStarted","Data":"e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed"}
Sep 11 15:55:54.439335 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:54.438757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns" event={"ID":"85831938-67dc-49ff-9d0f-9b62a3a6dc0d","Type":"ContainerStarted","Data":"465dc580c74798cb63e81a7ac0f9262b9d6b5f38983496205b89824415bf3a9d"}
Sep 11 15:55:54.439335 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:54.438806    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:55:54.440581 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:54.440550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpsls" event={"ID":"59b28b96-2421-41b3-8d69-842d2e47799b","Type":"ContainerStarted","Data":"4b0a9f88df13be90fea81fc917022c6cb117bdd175e729a48c3565fe06247b65"}
Sep 11 15:55:54.440720 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:54.440608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpsls" event={"ID":"59b28b96-2421-41b3-8d69-842d2e47799b","Type":"ContainerStarted","Data":"2e91891fd97ff43e99045ebc181703b9ab9046802e1f3cddcc3c5e6546c6c2ba"}
Sep 11 15:55:54.441762 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:54.441733    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtzf7" event={"ID":"852587b1-0a81-4241-a4bc-af2730e72a53","Type":"ContainerStarted","Data":"c12730ed6b384dc3e361bab652f570f3e7910408f2de3426190d49033eff17d6"}
Sep 11 15:55:54.461427 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:54.461370    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns" podStartSLOduration=29.461348296 podStartE2EDuration="29.461348296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:54.460487626 +0000 UTC m=+50.884305322" watchObservedRunningTime="2026-09-11 15:55:54.461348296 +0000 UTC m=+50.885165990"
Sep 11 15:55:55.447696 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:55.447657    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpsls" event={"ID":"59b28b96-2421-41b3-8d69-842d2e47799b","Type":"ContainerStarted","Data":"734ca633ec3e51abad6713ac2995e6a12218dbad94b3fe27c34b968c8e2ca480"}
Sep 11 15:55:56.453095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:56.453063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtzf7" event={"ID":"852587b1-0a81-4241-a4bc-af2730e72a53","Type":"ContainerStarted","Data":"9e5546c7b24af6b643471927fddf0c305d5691b3706fc9416f9b21733ac55108"}
Sep 11 15:55:56.453518 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:56.453123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jtzf7" event={"ID":"852587b1-0a81-4241-a4bc-af2730e72a53","Type":"ContainerStarted","Data":"3b94013de750e84116476f9de4fee856e5749742ade95c4be7e8cc1070d3e7b4"}
Sep 11 15:55:56.453518 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:56.453229    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:55:56.471762 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:56.471701    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jtzf7" podStartSLOduration=17.599556214 podStartE2EDuration="19.47167922s" totalImagesPullingTime="1.872123006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:56.469749724 +0000 UTC m=+52.893567455" watchObservedRunningTime="2026-09-11 15:55:56.47167922 +0000 UTC m=+52.895496912"
Sep 11 15:55:57.458491 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:57.458447    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpsls" event={"ID":"59b28b96-2421-41b3-8d69-842d2e47799b","Type":"ContainerStarted","Data":"8bc40984cbe76339c7cf47ee91d4e527cf819544b771443b79408c8ac5f13904"}
Sep 11 15:55:57.479545 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:57.479487    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jpsls" podStartSLOduration=17.95831289 podStartE2EDuration="20.479468619s" totalImagesPullingTime="2.521155729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:57.478083198 +0000 UTC m=+53.901900907" watchObservedRunningTime="2026-09-11 15:55:57.479468619 +0000 UTC m=+53.903286313"
Sep 11 15:55:59.652315 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.652280    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc6cb946f-j94ns"]
Sep 11 15:55:59.710376 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.710341    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67579569c8-v62fc"]
Sep 11 15:55:59.712813 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.712795    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.734715 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.734680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67579569c8-v62fc"]
Sep 11 15:55:59.880283 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-bound-sa-token\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.880442 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ab47786-bfd4-4230-983b-fa0d6098b424-image-registry-private-configuration\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.880442 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-registry-tls\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.880442 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ab47786-bfd4-4230-983b-fa0d6098b424-registry-certificates\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.880578 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4gbj\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-kube-api-access-h4gbj\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.880578 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ab47786-bfd4-4230-983b-fa0d6098b424-installation-pull-secrets\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.880578 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ab47786-bfd4-4230-983b-fa0d6098b424-ca-trust-extracted\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.880676 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.880617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ab47786-bfd4-4230-983b-fa0d6098b424-trusted-ca\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981073 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.980979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ab47786-bfd4-4230-983b-fa0d6098b424-installation-pull-secrets\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981073 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.981023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ab47786-bfd4-4230-983b-fa0d6098b424-ca-trust-extracted\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981073 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.981074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ab47786-bfd4-4230-983b-fa0d6098b424-trusted-ca\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981385 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.981097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-bound-sa-token\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981385 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.981192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ab47786-bfd4-4230-983b-fa0d6098b424-image-registry-private-configuration\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981385 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.981341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-registry-tls\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981536 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.981389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ab47786-bfd4-4230-983b-fa0d6098b424-registry-certificates\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.981589 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.981517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gbj\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-kube-api-access-h4gbj\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.982201 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.982179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ab47786-bfd4-4230-983b-fa0d6098b424-trusted-ca\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.982341 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.982307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ab47786-bfd4-4230-983b-fa0d6098b424-ca-trust-extracted\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.982796 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.982773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ab47786-bfd4-4230-983b-fa0d6098b424-registry-certificates\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.983643 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.983616    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ab47786-bfd4-4230-983b-fa0d6098b424-image-registry-private-configuration\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.983785 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.983769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ab47786-bfd4-4230-983b-fa0d6098b424-installation-pull-secrets\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.984353 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.984335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-registry-tls\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.990729 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.990708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-bound-sa-token\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:55:59.991015 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:55:59.990998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gbj\" (UniqueName: \"kubernetes.io/projected/6ab47786-bfd4-4230-983b-fa0d6098b424-kube-api-access-h4gbj\") pod \"image-registry-67579569c8-v62fc\" (UID: \"6ab47786-bfd4-4230-983b-fa0d6098b424\") " pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:56:00.022429 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:00.022391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:56:00.301572 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:00.301534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67579569c8-v62fc"]
Sep 11 15:56:00.305896 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:00.305869    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab47786_bfd4_4230_983b_fa0d6098b424.slice/crio-085ad5b6815223a3dec6cb4442af7f6380385da9fd7ca512886c27119558ed7d WatchSource:0}: Error finding container 085ad5b6815223a3dec6cb4442af7f6380385da9fd7ca512886c27119558ed7d: Status 404 returned error can't find the container with id 085ad5b6815223a3dec6cb4442af7f6380385da9fd7ca512886c27119558ed7d
Sep 11 15:56:00.472987 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:00.472947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67579569c8-v62fc" event={"ID":"6ab47786-bfd4-4230-983b-fa0d6098b424","Type":"ContainerStarted","Data":"ad473e4ff474a5b8e461775a9f40d69735f1ba77f70305f77c0b443876666dc9"}
Sep 11 15:56:00.473320 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:00.473005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67579569c8-v62fc" event={"ID":"6ab47786-bfd4-4230-983b-fa0d6098b424","Type":"ContainerStarted","Data":"085ad5b6815223a3dec6cb4442af7f6380385da9fd7ca512886c27119558ed7d"}
Sep 11 15:56:00.473320 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:00.473067    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:56:00.499774 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:00.499721    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67579569c8-v62fc" podStartSLOduration=1.499700157 podStartE2EDuration="1.499700157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:00.498801324 +0000 UTC m=+56.922619017" watchObservedRunningTime="2026-09-11 15:56:00.499700157 +0000 UTC m=+56.923517852"
Sep 11 15:56:03.328387 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.328350    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"]
Sep 11 15:56:03.331513 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.331494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"
Sep 11 15:56:03.334292 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.334266    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 15:56:03.334425 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.334357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rd4lk\""
Sep 11 15:56:03.342049 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.342025    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"]
Sep 11 15:56:03.366564 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.366535    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfjkn"
Sep 11 15:56:03.508331 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.508286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/75986d73-59ba-42ee-81e8-699504af6e0f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7fgnj\" (UID: \"75986d73-59ba-42ee-81e8-699504af6e0f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"
Sep 11 15:56:03.609073 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.608962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/75986d73-59ba-42ee-81e8-699504af6e0f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7fgnj\" (UID: \"75986d73-59ba-42ee-81e8-699504af6e0f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"
Sep 11 15:56:03.611456 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.611434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/75986d73-59ba-42ee-81e8-699504af6e0f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7fgnj\" (UID: \"75986d73-59ba-42ee-81e8-699504af6e0f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"
Sep 11 15:56:03.640454 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.640417    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"
Sep 11 15:56:03.910260 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:03.910074    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"]
Sep 11 15:56:03.912798 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:03.912772    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75986d73_59ba_42ee_81e8_699504af6e0f.slice/crio-1869dbb3363f74cce26e50b3c2c408675b6a357824b4361498ba35cbeea6f5b7 WatchSource:0}: Error finding container 1869dbb3363f74cce26e50b3c2c408675b6a357824b4361498ba35cbeea6f5b7: Status 404 returned error can't find the container with id 1869dbb3363f74cce26e50b3c2c408675b6a357824b4361498ba35cbeea6f5b7
Sep 11 15:56:04.487323 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:04.487282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj" event={"ID":"75986d73-59ba-42ee-81e8-699504af6e0f","Type":"ContainerStarted","Data":"1869dbb3363f74cce26e50b3c2c408675b6a357824b4361498ba35cbeea6f5b7"}
Sep 11 15:56:05.491966 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:05.491874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj" event={"ID":"75986d73-59ba-42ee-81e8-699504af6e0f","Type":"ContainerStarted","Data":"f4d8c54461452c5415aa6236c2cc22c67bafab52632684583ff399c94b9df388"}
Sep 11 15:56:05.492398 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:05.492099    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"
Sep 11 15:56:05.496898 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:05.496871    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj"
Sep 11 15:56:05.509304 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:05.509253    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7fgnj" podStartSLOduration=1.280721939 podStartE2EDuration="2.50921656s" totalImagesPullingTime="1.228494621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:05.509148583 +0000 UTC m=+61.932966278" watchObservedRunningTime="2026-09-11 15:56:05.50921656 +0000 UTC m=+61.933034284"
Sep 11 15:56:06.403224 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.403188    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"]
Sep 11 15:56:06.406312 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.406293    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.409543 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.409510    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 15:56:06.409997 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.409812    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 15:56:06.413406 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.410376    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 15:56:06.413406 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.412227    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 15:56:06.413406 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.412510    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 15:56:06.413406 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.412570    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jghnr\""
Sep 11 15:56:06.415911 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.415885    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"]
Sep 11 15:56:06.431774 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.431741    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f039024-5e0f-44f7-996d-f3fbbf3f3987-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.431961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.431825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.431961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.431887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.431961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.431924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnrl\" (UniqueName: \"kubernetes.io/projected/1f039024-5e0f-44f7-996d-f3fbbf3f3987-kube-api-access-xwnrl\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.461326 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.461297    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jtzf7"
Sep 11 15:56:06.532609 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.532570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.533094 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.532624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.533094 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.532660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnrl\" (UniqueName: \"kubernetes.io/projected/1f039024-5e0f-44f7-996d-f3fbbf3f3987-kube-api-access-xwnrl\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.533094 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.532738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f039024-5e0f-44f7-996d-f3fbbf3f3987-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.533094 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:06.532881    2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 11 15:56:06.533094 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:06.532954    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-tls podName:1f039024-5e0f-44f7-996d-f3fbbf3f3987 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:07.032933349 +0000 UTC m=+63.456751048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-tpbsc" (UID: "1f039024-5e0f-44f7-996d-f3fbbf3f3987") : secret "prometheus-operator-tls" not found
Sep 11 15:56:06.533537 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.533515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f039024-5e0f-44f7-996d-f3fbbf3f3987-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.535578 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.535544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:06.544486 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:06.544440    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnrl\" (UniqueName: \"kubernetes.io/projected/1f039024-5e0f-44f7-996d-f3fbbf3f3987-kube-api-access-xwnrl\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:07.036414 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:07.036373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:07.038746 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:07.038719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f039024-5e0f-44f7-996d-f3fbbf3f3987-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tpbsc\" (UID: \"1f039024-5e0f-44f7-996d-f3fbbf3f3987\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:07.319525 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:07.319428    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"
Sep 11 15:56:07.587288 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:07.587262    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc"]
Sep 11 15:56:07.589095 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:07.589064    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f039024_5e0f_44f7_996d_f3fbbf3f3987.slice/crio-9f63596385283b74fafb2a3cf8cf29ff26beedd0d8c5f5f85fb3d27091c70236 WatchSource:0}: Error finding container 9f63596385283b74fafb2a3cf8cf29ff26beedd0d8c5f5f85fb3d27091c70236: Status 404 returned error can't find the container with id 9f63596385283b74fafb2a3cf8cf29ff26beedd0d8c5f5f85fb3d27091c70236
Sep 11 15:56:08.503918 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.503879    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc" event={"ID":"1f039024-5e0f-44f7-996d-f3fbbf3f3987","Type":"ContainerStarted","Data":"9f63596385283b74fafb2a3cf8cf29ff26beedd0d8c5f5f85fb3d27091c70236"}
Sep 11 15:56:08.849323 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.849218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:56:08.849323 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.849308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:56:08.852426 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.852401    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 15:56:08.852551 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.852412    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 15:56:08.862416 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.862385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81f8ab6-37ce-477f-9eae-90441a29d9b3-original-pull-secret\") pod \"global-pull-secret-syncer-nnlqc\" (UID: \"c81f8ab6-37ce-477f-9eae-90441a29d9b3\") " pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:56:08.862416 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.862409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a20019-f1cd-4ee9-bd36-ba1d1e50a527-metrics-certs\") pod \"network-metrics-daemon-hspt8\" (UID: \"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527\") " pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:56:08.943691 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.943662    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zsbjs\""
Sep 11 15:56:08.947584 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.947560    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnlqc"
Sep 11 15:56:08.950420 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.950397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:56:08.950785 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.950766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hspt8"
Sep 11 15:56:08.953383 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.953366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 15:56:08.964686 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.964661    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 15:56:08.974958 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:08.974929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5wh\" (UniqueName: \"kubernetes.io/projected/d683e04d-faba-49bc-be34-f6e201a52d05-kube-api-access-5v5wh\") pod \"network-check-target-dvfp8\" (UID: \"d683e04d-faba-49bc-be34-f6e201a52d05\") " pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:56:09.223796 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.222797    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nnlqc"]
Sep 11 15:56:09.226149 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:09.226097    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81f8ab6_37ce_477f_9eae_90441a29d9b3.slice/crio-3835df56feed1c595a593aa54e29d680d29839261f55490e86d1907c23ec17e4 WatchSource:0}: Error finding container 3835df56feed1c595a593aa54e29d680d29839261f55490e86d1907c23ec17e4: Status 404 returned error can't find the container with id 3835df56feed1c595a593aa54e29d680d29839261f55490e86d1907c23ec17e4
Sep 11 15:56:09.249425 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.249393    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hspt8"]
Sep 11 15:56:09.252635 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:09.252606    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a20019_f1cd_4ee9_bd36_ba1d1e50a527.slice/crio-bb5b6fc9b0979f8735ff86da1bfe8ea52119cb7591f655d24b120de130bce906 WatchSource:0}: Error finding container bb5b6fc9b0979f8735ff86da1bfe8ea52119cb7591f655d24b120de130bce906: Status 404 returned error can't find the container with id bb5b6fc9b0979f8735ff86da1bfe8ea52119cb7591f655d24b120de130bce906
Sep 11 15:56:09.254991 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.254955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lsspg\""
Sep 11 15:56:09.262512 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.262493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:56:09.508939 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.508904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc" event={"ID":"1f039024-5e0f-44f7-996d-f3fbbf3f3987","Type":"ContainerStarted","Data":"94310fa5351b9819219022f92bb248f21775fd6161fd524c289affc34d555c60"}
Sep 11 15:56:09.509151 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.508968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc" event={"ID":"1f039024-5e0f-44f7-996d-f3fbbf3f3987","Type":"ContainerStarted","Data":"64330613ba1f9c5ba716d950de6d0663779efb5b9ea9b17fe3584852ac4d3412"}
Sep 11 15:56:09.510053 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.510029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hspt8" event={"ID":"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527","Type":"ContainerStarted","Data":"bb5b6fc9b0979f8735ff86da1bfe8ea52119cb7591f655d24b120de130bce906"}
Sep 11 15:56:09.511008 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.510989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nnlqc" event={"ID":"c81f8ab6-37ce-477f-9eae-90441a29d9b3","Type":"ContainerStarted","Data":"3835df56feed1c595a593aa54e29d680d29839261f55490e86d1907c23ec17e4"}
Sep 11 15:56:09.528892 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.528837    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tpbsc" podStartSLOduration=2.144108933 podStartE2EDuration="3.528819361s" totalImagesPullingTime="1.384710428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:09.527751864 +0000 UTC m=+65.951569579" watchObservedRunningTime="2026-09-11 15:56:09.528819361 +0000 UTC m=+65.952637056"
Sep 11 15:56:09.532293 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.532265    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvfp8"]
Sep 11 15:56:09.536081 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:09.536047    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd683e04d_faba_49bc_be34_f6e201a52d05.slice/crio-7f8bdd5e9c96d0998db834b40f37b6edc227c993f711bb20ce776265b6221c88 WatchSource:0}: Error finding container 7f8bdd5e9c96d0998db834b40f37b6edc227c993f711bb20ce776265b6221c88: Status 404 returned error can't find the container with id 7f8bdd5e9c96d0998db834b40f37b6edc227c993f711bb20ce776265b6221c88
Sep 11 15:56:09.658943 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.658903    2559 patch_prober.go:28] interesting pod/image-registry-7fc6cb946f-j94ns container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 15:56:09.659107 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:09.658970    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns" podUID="85831938-67dc-49ff-9d0f-9b62a3a6dc0d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 15:56:10.519092 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:10.519007    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvfp8" event={"ID":"d683e04d-faba-49bc-be34-f6e201a52d05","Type":"ContainerStarted","Data":"7f8bdd5e9c96d0998db834b40f37b6edc227c993f711bb20ce776265b6221c88"}
Sep 11 15:56:11.526942 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.526907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hspt8" event={"ID":"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527","Type":"ContainerStarted","Data":"4e06d94f59e0f0f11c3e58a1279822ee1d8e2a51aa3989c29a70616d5c21071b"}
Sep 11 15:56:11.527424 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.526964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hspt8" event={"ID":"e4a20019-f1cd-4ee9-bd36-ba1d1e50a527","Type":"ContainerStarted","Data":"9d4e3baeb61607d23161b5a4c602e8ee4d0d8303fb94e18be8aa49fc3933b0c9"}
Sep 11 15:56:11.785328 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.783420    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hspt8" podStartSLOduration=66.606297091 podStartE2EDuration="1m7.783395504s" totalImagesPullingTime="1.177098413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:11.558533391 +0000 UTC m=+67.982351111" watchObservedRunningTime="2026-09-11 15:56:11.783395504 +0000 UTC m=+68.207213198"
Sep 11 15:56:11.785328 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.784222    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"]
Sep 11 15:56:11.790686 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.789635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.797257 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.796799    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 15:56:11.797257 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.797079    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bfjzr\""
Sep 11 15:56:11.797473 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.797304    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 15:56:11.811699 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.811668    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"]
Sep 11 15:56:11.820447 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.819308    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g9vvc"]
Sep 11 15:56:11.829251 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.825582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.829474 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.829393    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 15:56:11.832264 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.830359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9j5tf\""
Sep 11 15:56:11.832264 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.830639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 15:56:11.832264 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.831116    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 15:56:11.877377 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.877377 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-tls\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-textfile\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877451    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-metrics-client-ca\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhjg\" (UniqueName: \"kubernetes.io/projected/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-kube-api-access-pkhjg\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59rf\" (UniqueName: \"kubernetes.io/projected/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-kube-api-access-c59rf\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-wtmp\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.877631 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-sys\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.878106 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.877676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-root\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978409 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.978409 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-tls\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978409 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.978716 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-textfile\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978716 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-metrics-client-ca\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978716 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.978716 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978716 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978726    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhjg\" (UniqueName: \"kubernetes.io/projected/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-kube-api-access-pkhjg\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59rf\" (UniqueName: \"kubernetes.io/projected/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-kube-api-access-c59rf\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.978961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-wtmp\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-sys\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.978961 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-root\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.979199 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.978960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-root\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.979554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-textfile\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.979608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.980171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.980187    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-wtmp\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.980260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-sys\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:11.980277    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:11.980338    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-tls podName:5cd34ec5-28af-4e7b-9816-b3aec953d6dc nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.48031663 +0000 UTC m=+68.904134308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-tls") pod "node-exporter-g9vvc" (UID: "5cd34ec5-28af-4e7b-9816-b3aec953d6dc") : secret "node-exporter-tls" not found
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:11.980567    2559 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 15:56:11.980702 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:11.980607    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-tls podName:43dc71f8-7bc8-4ccc-99fd-112a5798b4fc nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.480594392 +0000 UTC m=+68.904412069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-q52hv" (UID: "43dc71f8-7bc8-4ccc-99fd-112a5798b4fc") : secret "openshift-state-metrics-tls" not found
Sep 11 15:56:11.986601 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.980761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-metrics-client-ca\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.986601 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.986390    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.986601 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.986502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:11.992874 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.992842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhjg\" (UniqueName: \"kubernetes.io/projected/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-kube-api-access-pkhjg\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:11.994496 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:11.994380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59rf\" (UniqueName: \"kubernetes.io/projected/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-kube-api-access-c59rf\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:12.483975 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:12.483936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-tls\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:12.484160 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:12.483993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:12.486810 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:12.486779    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5cd34ec5-28af-4e7b-9816-b3aec953d6dc-node-exporter-tls\") pod \"node-exporter-g9vvc\" (UID: \"5cd34ec5-28af-4e7b-9816-b3aec953d6dc\") " pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:12.486933 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:12.486826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43dc71f8-7bc8-4ccc-99fd-112a5798b4fc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q52hv\" (UID: \"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:12.718013 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:12.717976    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"
Sep 11 15:56:12.745088 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:12.745055    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g9vvc"
Sep 11 15:56:13.755841 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.755800    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5fb98cd666-rflpk"]
Sep 11 15:56:13.780484 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.780445    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fb98cd666-rflpk"]
Sep 11 15:56:13.780689 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.780580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.783822 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.783590    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 15:56:13.783822 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.783829    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 15:56:13.784046 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.784019    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 15:56:13.784115 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.784079    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5glpd\""
Sep 11 15:56:13.784318 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.784023    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 15:56:13.784318 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.784309    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 15:56:13.784560 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.784540    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f0m08gq17rt39\""
Sep 11 15:56:13.796604 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.796574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.796753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.796625    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-tls\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.796753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.796666    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.796753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.796705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.796753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.796722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.796753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.796746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59sv\" (UniqueName: \"kubernetes.io/projected/9a13f4b3-6021-4081-8951-3b3f29e2c384-kube-api-access-l59sv\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.797003 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.796795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-grpc-tls\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.797058 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.797010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a13f4b3-6021-4081-8951-3b3f29e2c384-metrics-client-ca\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898431 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-grpc-tls\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898615 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a13f4b3-6021-4081-8951-3b3f29e2c384-metrics-client-ca\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898615 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898615 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-tls\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898771 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898771 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898771 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.898942 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.898791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59sv\" (UniqueName: \"kubernetes.io/projected/9a13f4b3-6021-4081-8951-3b3f29e2c384-kube-api-access-l59sv\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.899442 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.899303    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a13f4b3-6021-4081-8951-3b3f29e2c384-metrics-client-ca\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.901628 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.901604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.901743 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.901671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.901886 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.901853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-tls\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.901886 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.901877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.902106 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.902063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.902575 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.902558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a13f4b3-6021-4081-8951-3b3f29e2c384-secret-grpc-tls\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:13.907339 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:13.907312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59sv\" (UniqueName: \"kubernetes.io/projected/9a13f4b3-6021-4081-8951-3b3f29e2c384-kube-api-access-l59sv\") pod \"thanos-querier-5fb98cd666-rflpk\" (UID: \"9a13f4b3-6021-4081-8951-3b3f29e2c384\") " pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:14.093563 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:14.093467    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:15.070266 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.070108    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv"]
Sep 11 15:56:15.072975 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:15.072928    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43dc71f8_7bc8_4ccc_99fd_112a5798b4fc.slice/crio-ec485023bd2ef01f38590e77c784f9f16b9e4b3931813eb974bed019ea9077b0 WatchSource:0}: Error finding container ec485023bd2ef01f38590e77c784f9f16b9e4b3931813eb974bed019ea9077b0: Status 404 returned error can't find the container with id ec485023bd2ef01f38590e77c784f9f16b9e4b3931813eb974bed019ea9077b0
Sep 11 15:56:15.089727 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.089697    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fb98cd666-rflpk"]
Sep 11 15:56:15.091551 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:15.091522    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a13f4b3_6021_4081_8951_3b3f29e2c384.slice/crio-82c5fee4409936d228fef54f1a2dff356c67274ad09d1b3fbaff2d1e5945eba8 WatchSource:0}: Error finding container 82c5fee4409936d228fef54f1a2dff356c67274ad09d1b3fbaff2d1e5945eba8: Status 404 returned error can't find the container with id 82c5fee4409936d228fef54f1a2dff356c67274ad09d1b3fbaff2d1e5945eba8
Sep 11 15:56:15.551263 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.549161    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv" event={"ID":"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc","Type":"ContainerStarted","Data":"4609793b184116c9e493c4de9ef93a8264e12666bb6cb46bed290f201b8453b5"}
Sep 11 15:56:15.551263 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.549302    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv" event={"ID":"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc","Type":"ContainerStarted","Data":"43e2f932de0a1ce5d8360ac3641ec0275abab935de8aa6a2351649ca9a18da2a"}
Sep 11 15:56:15.551263 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.549344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv" event={"ID":"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc","Type":"ContainerStarted","Data":"ec485023bd2ef01f38590e77c784f9f16b9e4b3931813eb974bed019ea9077b0"}
Sep 11 15:56:15.554126 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.554089    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nnlqc" event={"ID":"c81f8ab6-37ce-477f-9eae-90441a29d9b3","Type":"ContainerStarted","Data":"767d6d6b6e97f8cfd21bc0481bd80219b617cffbf77909df62e6d6f90627a315"}
Sep 11 15:56:15.555930 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.555903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvfp8" event={"ID":"d683e04d-faba-49bc-be34-f6e201a52d05","Type":"ContainerStarted","Data":"33525c76ba709a93df6be9d03957d98054ade0dfa390fdc7d90f2b32ea3eaffd"}
Sep 11 15:56:15.556119 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.556097    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:56:15.557457 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.557432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" event={"ID":"9a13f4b3-6021-4081-8951-3b3f29e2c384","Type":"ContainerStarted","Data":"82c5fee4409936d228fef54f1a2dff356c67274ad09d1b3fbaff2d1e5945eba8"}
Sep 11 15:56:15.558828 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.558807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9vvc" event={"ID":"5cd34ec5-28af-4e7b-9816-b3aec953d6dc","Type":"ContainerStarted","Data":"72d94ed301c4352caff8f57cd237f8fc6f140ef65054d8ba352b9710544a162d"}
Sep 11 15:56:15.570479 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.570417    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nnlqc" podStartSLOduration=65.997827991 podStartE2EDuration="1m11.570394884s" totalImagesPullingTime="5.572566893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:15.569685655 +0000 UTC m=+71.993503350" watchObservedRunningTime="2026-09-11 15:56:15.570394884 +0000 UTC m=+71.994212581"
Sep 11 15:56:15.587619 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:15.587550    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dvfp8" podStartSLOduration=66.332932365 podStartE2EDuration="1m11.587528221s" totalImagesPullingTime="5.254595856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:15.586689897 +0000 UTC m=+72.010507614" watchObservedRunningTime="2026-09-11 15:56:15.587528221 +0000 UTC m=+72.011345914"
Sep 11 15:56:16.551764 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.551726    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"]
Sep 11 15:56:16.565627 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.565595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:16.565807 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.565647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"]
Sep 11 15:56:16.569626 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.569591    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cd34ec5-28af-4e7b-9816-b3aec953d6dc" containerID="e3c5f9a4589ecd2c110fc4c86b6162a4eb8aeb44e4cd697324bae0a76be25dd1" exitCode=0
Sep 11 15:56:16.569850 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.569826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9vvc" event={"ID":"5cd34ec5-28af-4e7b-9816-b3aec953d6dc","Type":"ContainerDied","Data":"e3c5f9a4589ecd2c110fc4c86b6162a4eb8aeb44e4cd697324bae0a76be25dd1"}
Sep 11 15:56:16.570041 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.570018    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dd98d\""
Sep 11 15:56:16.570549 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.570489    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 15:56:16.627934 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.627892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2xc\" (UniqueName: \"kubernetes.io/projected/ef47f679-d474-400d-83a7-c760fb7fd96f-kube-api-access-rk2xc\") pod \"monitoring-plugin-5d85778b9d-rnlnc\" (UID: \"ef47f679-d474-400d-83a7-c760fb7fd96f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:16.628118 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.628023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef47f679-d474-400d-83a7-c760fb7fd96f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rnlnc\" (UID: \"ef47f679-d474-400d-83a7-c760fb7fd96f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:16.729880 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.729836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2xc\" (UniqueName: \"kubernetes.io/projected/ef47f679-d474-400d-83a7-c760fb7fd96f-kube-api-access-rk2xc\") pod \"monitoring-plugin-5d85778b9d-rnlnc\" (UID: \"ef47f679-d474-400d-83a7-c760fb7fd96f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:16.730066 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.729907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef47f679-d474-400d-83a7-c760fb7fd96f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rnlnc\" (UID: \"ef47f679-d474-400d-83a7-c760fb7fd96f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:16.730066 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:16.730031    2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 15:56:16.730185 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:16.730085    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef47f679-d474-400d-83a7-c760fb7fd96f-monitoring-plugin-cert podName:ef47f679-d474-400d-83a7-c760fb7fd96f nodeName:}" failed. No retries permitted until 2026-09-11 15:56:17.230068575 +0000 UTC m=+73.653886248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ef47f679-d474-400d-83a7-c760fb7fd96f-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-rnlnc" (UID: "ef47f679-d474-400d-83a7-c760fb7fd96f") : secret "monitoring-plugin-cert" not found
Sep 11 15:56:16.744192 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:16.740458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2xc\" (UniqueName: \"kubernetes.io/projected/ef47f679-d474-400d-83a7-c760fb7fd96f-kube-api-access-rk2xc\") pod \"monitoring-plugin-5d85778b9d-rnlnc\" (UID: \"ef47f679-d474-400d-83a7-c760fb7fd96f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:17.234418 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.234378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef47f679-d474-400d-83a7-c760fb7fd96f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rnlnc\" (UID: \"ef47f679-d474-400d-83a7-c760fb7fd96f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:17.237162 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.237130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef47f679-d474-400d-83a7-c760fb7fd96f-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rnlnc\" (UID: \"ef47f679-d474-400d-83a7-c760fb7fd96f\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:17.478950 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.478929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:17.577321 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.577283    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" event={"ID":"9a13f4b3-6021-4081-8951-3b3f29e2c384","Type":"ContainerStarted","Data":"1b82182131a73abaa4999aa583ec447b32987450430706df843f46c60f00fd80"}
Sep 11 15:56:17.577777 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.577379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" event={"ID":"9a13f4b3-6021-4081-8951-3b3f29e2c384","Type":"ContainerStarted","Data":"363fe52de7f05e720a7db4a87ea93603a2daac0763a7d5db8cc0022491b92b71"}
Sep 11 15:56:17.579976 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.579948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9vvc" event={"ID":"5cd34ec5-28af-4e7b-9816-b3aec953d6dc","Type":"ContainerStarted","Data":"344e09df073fd720bf46bac6f1b41589bdeec9a646b1d452ab4509867f3d2436"}
Sep 11 15:56:17.580098 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.580011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g9vvc" event={"ID":"5cd34ec5-28af-4e7b-9816-b3aec953d6dc","Type":"ContainerStarted","Data":"e0ba25ec407a1de770119fb339c98a4ee511a98fd3b11eae0d9a34432d6c9557"}
Sep 11 15:56:17.581927 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.581903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv" event={"ID":"43dc71f8-7bc8-4ccc-99fd-112a5798b4fc","Type":"ContainerStarted","Data":"333fe277b425a05834625e303b058e3dcda9fff5686b6527a6d80edc97012683"}
Sep 11 15:56:17.603266 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.603194    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g9vvc" podStartSLOduration=5.192491077 podStartE2EDuration="6.603173068s" totalImagesPullingTime="1.319169935s" totalInitContainerRuntime="91.512056ms" isStatefulPod=true podCreationTimestamp="2026-09-11 15:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:17.601845847 +0000 UTC m=+74.025663568" watchObservedRunningTime="2026-09-11 15:56:17.603173068 +0000 UTC m=+74.026990766"
Sep 11 15:56:17.623954 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.623898    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q52hv" podStartSLOduration=4.528721193 podStartE2EDuration="6.623881761s" totalImagesPullingTime="2.095160568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:17.623119152 +0000 UTC m=+74.046936870" watchObservedRunningTime="2026-09-11 15:56:17.623881761 +0000 UTC m=+74.047699455"
Sep 11 15:56:17.771322 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.771291    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"]
Sep 11 15:56:17.774455 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:56:17.774413    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef47f679_d474_400d_83a7_c760fb7fd96f.slice/crio-b524a43b08ecddddce8ee855ab1a2832b67eef7a3c11f12b15e62866fcbb5859 WatchSource:0}: Error finding container b524a43b08ecddddce8ee855ab1a2832b67eef7a3c11f12b15e62866fcbb5859: Status 404 returned error can't find the container with id b524a43b08ecddddce8ee855ab1a2832b67eef7a3c11f12b15e62866fcbb5859
Sep 11 15:56:17.976918 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.976878    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:56:17.980055 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.980028    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:17.983582 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.983554    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r78wg\""
Sep 11 15:56:17.983996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.983827    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 15:56:17.983996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.983853    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 15:56:17.983996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.983868    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1brm7orfpcpsa\""
Sep 11 15:56:17.983996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.983891    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 15:56:17.983996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.983853    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 15:56:17.985402 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985358    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 15:56:17.985519 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985445    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 15:56:17.985519 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985489    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 15:56:17.985869 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985722    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 15:56:17.985869 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985747    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 15:56:17.985869 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985812    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 15:56:17.985869 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985815    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 15:56:17.986156 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985932    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 15:56:17.986156 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985965    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 15:56:17.986156 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.985995    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 15:56:17.987409 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.987391    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 15:56:17.998599 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:17.998572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:56:18.040684 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040684 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040635    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040684 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-web-config\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040798    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config-out\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.040955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040937    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.040978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5fj\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-kube-api-access-zw5fj\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.041348 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.041285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142400 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142400 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-web-config\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142457    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config-out\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.142679 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143101 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143101 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143101 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143101 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143101 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143101 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142883    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5fj\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-kube-api-access-zw5fj\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143101 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.142934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143699 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.143675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143801 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.143720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143801 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.143751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.143801 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.143778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.144790 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.144757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.146160 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.145318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.146160 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.145429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.146160 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.146119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.146393 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.146362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.147123 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.147093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.148038 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.148010    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.148431 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.148407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.148514 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.148412    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config-out\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.149717 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.149690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.149809 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.149727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.150175 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.150154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.150298 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.150279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.151001 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.150981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.151147 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.151064    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.151332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.151316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-web-config\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.152041 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.152026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.153264 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.153230    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5fj\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-kube-api-access-zw5fj\") pod \"prometheus-k8s-0\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.293626 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.293535    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:18.580643 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.580617    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:56:18.592042 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.592008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" event={"ID":"9a13f4b3-6021-4081-8951-3b3f29e2c384","Type":"ContainerStarted","Data":"3d36dca2e63e74936d6f4cfe09c07500285a791e5e6ff6418353a168007ec055"}
Sep 11 15:56:18.593602 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:18.593549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc" event={"ID":"ef47f679-d474-400d-83a7-c760fb7fd96f","Type":"ContainerStarted","Data":"b524a43b08ecddddce8ee855ab1a2832b67eef7a3c11f12b15e62866fcbb5859"}
Sep 11 15:56:19.599766 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.599671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerStarted","Data":"30c3f6fe1fffdfbbff493b2b1f1e4abfe25ebed0e6b0d150dc32777c8f61b4a9"}
Sep 11 15:56:19.601726 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.601690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc" event={"ID":"ef47f679-d474-400d-83a7-c760fb7fd96f","Type":"ContainerStarted","Data":"9c79b548ed215ae7ee7d9acaf272812e707e061d6dbac9e6607df881cafdf2c3"}
Sep 11 15:56:19.601874 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.601750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:19.606095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.605718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" event={"ID":"9a13f4b3-6021-4081-8951-3b3f29e2c384","Type":"ContainerStarted","Data":"30e4c2a87411154f4996eda5d2d37f5ccd86a0f6c5f1ecac8280c88b5cb2ae6f"}
Sep 11 15:56:19.606095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.605788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" event={"ID":"9a13f4b3-6021-4081-8951-3b3f29e2c384","Type":"ContainerStarted","Data":"58f74a6c3ed30ad1df3a3723f6b9ff9edb4632cdf237fc4320d4ce87458c1c31"}
Sep 11 15:56:19.606095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.605826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" event={"ID":"9a13f4b3-6021-4081-8951-3b3f29e2c384","Type":"ContainerStarted","Data":"336877aaf0b6ba01c55192e3271844afafec4bb63c98f4ac62e4c8a0bcdbd675"}
Sep 11 15:56:19.606095 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.605869    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:19.608340 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.608319    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc"
Sep 11 15:56:19.621808 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.621759    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnlnc" podStartSLOduration=2.147302975 podStartE2EDuration="3.621738412s" totalImagesPullingTime="1.474435437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:19.619834246 +0000 UTC m=+76.043651960" watchObservedRunningTime="2026-09-11 15:56:19.621738412 +0000 UTC m=+76.045556108"
Sep 11 15:56:19.644454 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.644398    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk" podStartSLOduration=3.416522664 podStartE2EDuration="6.644372597s" totalImagesPullingTime="3.227849933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:19.642544442 +0000 UTC m=+76.066362150" watchObservedRunningTime="2026-09-11 15:56:19.644372597 +0000 UTC m=+76.068190293"
Sep 11 15:56:19.657320 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:19.657289    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:56:20.611045 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:20.611006    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964" exitCode=0
Sep 11 15:56:20.611559 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:20.611101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"}
Sep 11 15:56:21.482481 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:21.482447    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67579569c8-v62fc"
Sep 11 15:56:24.634070 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.634027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerStarted","Data":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"}
Sep 11 15:56:24.634467 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.634120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerStarted","Data":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"}
Sep 11 15:56:24.634467 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.634167    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerStarted","Data":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"}
Sep 11 15:56:24.634467 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.634204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerStarted","Data":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"}
Sep 11 15:56:24.634467 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.634268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerStarted","Data":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"}
Sep 11 15:56:24.634467 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.634303    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerStarted","Data":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"}
Sep 11 15:56:24.670904 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.670836    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.337488361 podStartE2EDuration="7.670812507s" totalImagesPullingTime="4.314680762s" totalInitContainerRuntime="18.643384ms" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:24.668964497 +0000 UTC m=+81.092782209" watchObservedRunningTime="2026-09-11 15:56:24.670812507 +0000 UTC m=+81.094630199"
Sep 11 15:56:24.672199 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.672126    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns" podUID="85831938-67dc-49ff-9d0f-9b62a3a6dc0d" containerName="registry" containerID="cri-o://e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed" gracePeriod=30
Sep 11 15:56:24.916215 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:24.916176    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:56:25.011814 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.011766    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm2vv\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-kube-api-access-sm2vv\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.011814 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.011824    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-image-registry-private-configuration\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.012066 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.011857    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-trusted-ca\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.012066 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.011895    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-ca-trust-extracted\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.012066 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.011915    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-bound-sa-token\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.012066 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.011930    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-certificates\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.012066 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.011984    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.012066 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.012007    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-installation-pull-secrets\") pod \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\" (UID: \"85831938-67dc-49ff-9d0f-9b62a3a6dc0d\") "
Sep 11 15:56:25.012459 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.012429    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:56:25.012656 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.012626    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:56:25.014543 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.014501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:56:25.014660 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.014571    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-kube-api-access-sm2vv" (OuterVolumeSpecName: "kube-api-access-sm2vv") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "kube-api-access-sm2vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:56:25.014660 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.014603    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:56:25.014827 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.014809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:56:25.014864 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.014842    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:56:25.020715 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.020690    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "85831938-67dc-49ff-9d0f-9b62a3a6dc0d" (UID: "85831938-67dc-49ff-9d0f-9b62a3a6dc0d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:56:25.112694 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112649    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.112694 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112687    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-installation-pull-secrets\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.112694 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112698    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2vv\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-kube-api-access-sm2vv\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.112694 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112709    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-image-registry-private-configuration\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.112963 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112719    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-trusted-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.112963 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112727    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-ca-trust-extracted\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.112963 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112735    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-bound-sa-token\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.112963 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.112744    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85831938-67dc-49ff-9d0f-9b62a3a6dc0d-registry-certificates\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:56:25.619096 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.619066    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5fb98cd666-rflpk"
Sep 11 15:56:25.639830 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.639798    2559 generic.go:356] "Generic (PLEG): container finished" podID="85831938-67dc-49ff-9d0f-9b62a3a6dc0d" containerID="e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed" exitCode=0
Sep 11 15:56:25.640343 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.639863    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns"
Sep 11 15:56:25.640343 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.639889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns" event={"ID":"85831938-67dc-49ff-9d0f-9b62a3a6dc0d","Type":"ContainerDied","Data":"e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed"}
Sep 11 15:56:25.640343 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.639940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc6cb946f-j94ns" event={"ID":"85831938-67dc-49ff-9d0f-9b62a3a6dc0d","Type":"ContainerDied","Data":"465dc580c74798cb63e81a7ac0f9262b9d6b5f38983496205b89824415bf3a9d"}
Sep 11 15:56:25.640343 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.639960    2559 scope.go:117] "RemoveContainer" containerID="e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed"
Sep 11 15:56:25.649976 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.649956    2559 scope.go:117] "RemoveContainer" containerID="e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed"
Sep 11 15:56:25.650310 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:56:25.650276    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed\": container with ID starting with e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed not found: ID does not exist" containerID="e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed"
Sep 11 15:56:25.650392 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.650320    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed"} err="failed to get container status \"e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed\": rpc error: code = NotFound desc = could not find container \"e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed\": container with ID starting with e4e656cfdd73d86501aaf2a53e3b45749cd8fe4a22ace20540fd21af80e651ed not found: ID does not exist"
Sep 11 15:56:25.663103 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.663073    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc6cb946f-j94ns"]
Sep 11 15:56:25.672292 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:25.670383    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fc6cb946f-j94ns"]
Sep 11 15:56:26.034561 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:26.034524    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85831938-67dc-49ff-9d0f-9b62a3a6dc0d" path="/var/lib/kubelet/pods/85831938-67dc-49ff-9d0f-9b62a3a6dc0d/volumes"
Sep 11 15:56:28.294585 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:28.294531    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:56:46.573099 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:56:46.573060    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dvfp8"
Sep 11 15:57:18.294759 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:18.294693    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:18.314905 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:18.314876    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:18.876681 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:18.876652    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:36.433732 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.433695    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:57:36.434297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.434028    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="prometheus" containerID="cri-o://24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd" gracePeriod=600
Sep 11 15:57:36.434297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.434048    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy" containerID="cri-o://875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903" gracePeriod=600
Sep 11 15:57:36.434297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.434104    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="thanos-sidecar" containerID="cri-o://a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205" gracePeriod=600
Sep 11 15:57:36.434297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.434119    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="config-reloader" containerID="cri-o://c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141" gracePeriod=600
Sep 11 15:57:36.434297 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.434223    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-web" containerID="cri-o://528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532" gracePeriod=600
Sep 11 15:57:36.434566 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.434315    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a" gracePeriod=600
Sep 11 15:57:36.704769 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.704740    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:36.813626 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813591    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-kube-rbac-proxy\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813626 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813635    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-1\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813653    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config-out\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-tls-assets\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813698    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-db\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813715    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-metrics-client-ca\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813734    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-web-config\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813777    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-0\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813801    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-trusted-ca-bundle\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813826    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.813862 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813859    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw5fj\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-kube-api-access-zw5fj\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813921    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-thanos-prometheus-http-client-file\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813940    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-2\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813960    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-serving-certs-ca-bundle\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.813983    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.814013    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-metrics-client-certs\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.814032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-kubelet-serving-ca-bundle\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.814067    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-tls\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814349 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.814083    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-grpc-tls\") pod \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\" (UID: \"7bd32269-f6ae-4c91-b417-1bc2b8eb4620\") "
Sep 11 15:57:36.814720 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.814474    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:57:36.814776 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.814751    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:57:36.815141 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.815105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:57:36.815837 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.815806    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:57:36.817267 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.816894    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.817267 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.816995    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:57:36.817267 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.817217    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:57:36.817486 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.817316    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:57:36.817486 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.817390    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.817486 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.817413    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:57:36.817652 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.817487    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config-out" (OuterVolumeSpecName: "config-out") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:57:36.817870 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.817848    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:57:36.818381 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.818352    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config" (OuterVolumeSpecName: "config") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.818690 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.818655    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-kube-api-access-zw5fj" (OuterVolumeSpecName: "kube-api-access-zw5fj") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "kube-api-access-zw5fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:57:36.819278 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.819250    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.819376 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.819359    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.819460 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.819443    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.819722 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.819698    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.819763 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.819717    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.827991 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.827959    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-web-config" (OuterVolumeSpecName: "web-config") pod "7bd32269-f6ae-4c91-b417-1bc2b8eb4620" (UID: "7bd32269-f6ae-4c91-b417-1bc2b8eb4620"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:57:36.914908 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914847    2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.914908 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914898    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.914908 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914908    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.914908 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914919    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914935    2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-metrics-client-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914953    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914962    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914971    2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-grpc-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914979    2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-kube-rbac-proxy\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914987    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.914995    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config-out\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915003    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-tls-assets\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915012    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-db\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915019    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-configmap-metrics-client-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915029    2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915037    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-web-config\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915044    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915053    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915062    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.915180 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.915070    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5fj\" (UniqueName: \"kubernetes.io/projected/7bd32269-f6ae-4c91-b417-1bc2b8eb4620-kube-api-access-zw5fj\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 15:57:36.931919 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931881    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a" exitCode=0
Sep 11 15:57:36.931919 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931911    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903" exitCode=0
Sep 11 15:57:36.931919 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931920    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532" exitCode=0
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931928    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205" exitCode=0
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931938    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141" exitCode=0
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931947    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd" exitCode=0
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"}
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.931977    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.932035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"}
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.932063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"}
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.932089    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"}
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.932109    2559 scope.go:117] "RemoveContainer" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.932114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"}
Sep 11 15:57:36.932137 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.932140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"}
Sep 11 15:57:36.932563 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.932166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7bd32269-f6ae-4c91-b417-1bc2b8eb4620","Type":"ContainerDied","Data":"30c3f6fe1fffdfbbff493b2b1f1e4abfe25ebed0e6b0d150dc32777c8f61b4a9"}
Sep 11 15:57:36.940290 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.940265    2559 scope.go:117] "RemoveContainer" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.947138 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.947119    2559 scope.go:117] "RemoveContainer" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.953869 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.953846    2559 scope.go:117] "RemoveContainer" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.958601 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.958573    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:57:36.962268 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.962230    2559 scope.go:117] "RemoveContainer" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.963142 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.963121    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:57:36.969183 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.969166    2559 scope.go:117] "RemoveContainer" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.976455 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.976433    2559 scope.go:117] "RemoveContainer" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.986162 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.986144    2559 scope.go:117] "RemoveContainer" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.986500 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:57:36.986479    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": container with ID starting with d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a not found: ID does not exist" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.986549 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.986512    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"} err="failed to get container status \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": rpc error: code = NotFound desc = could not find container \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": container with ID starting with d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a not found: ID does not exist"
Sep 11 15:57:36.986549 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.986532    2559 scope.go:117] "RemoveContainer" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.986788 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:57:36.986771    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": container with ID starting with 875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903 not found: ID does not exist" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.986845 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.986799    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"} err="failed to get container status \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": rpc error: code = NotFound desc = could not find container \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": container with ID starting with 875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903 not found: ID does not exist"
Sep 11 15:57:36.986845 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.986822    2559 scope.go:117] "RemoveContainer" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.987067 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:57:36.987051    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": container with ID starting with 528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532 not found: ID does not exist" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.987105 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.987071    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"} err="failed to get container status \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": rpc error: code = NotFound desc = could not find container \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": container with ID starting with 528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532 not found: ID does not exist"
Sep 11 15:57:36.987105 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.987088    2559 scope.go:117] "RemoveContainer" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.987445 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:57:36.987429    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": container with ID starting with a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205 not found: ID does not exist" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.987494 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.987447    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"} err="failed to get container status \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": rpc error: code = NotFound desc = could not find container \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": container with ID starting with a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205 not found: ID does not exist"
Sep 11 15:57:36.987494 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.987460    2559 scope.go:117] "RemoveContainer" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.987692 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:57:36.987676    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": container with ID starting with c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141 not found: ID does not exist" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.987753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.987708    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"} err="failed to get container status \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": rpc error: code = NotFound desc = could not find container \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": container with ID starting with c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141 not found: ID does not exist"
Sep 11 15:57:36.987753 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.987731    2559 scope.go:117] "RemoveContainer" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.988020 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:57:36.988004    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": container with ID starting with 24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd not found: ID does not exist" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.988065 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988023    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"} err="failed to get container status \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": rpc error: code = NotFound desc = could not find container \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": container with ID starting with 24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd not found: ID does not exist"
Sep 11 15:57:36.988065 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988038    2559 scope.go:117] "RemoveContainer" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.988302 ip-10-0-129-137 kubenswrapper[2559]: E0911 15:57:36.988273    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": container with ID starting with bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964 not found: ID does not exist" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.988350 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988308    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"} err="failed to get container status \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": rpc error: code = NotFound desc = could not find container \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": container with ID starting with bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964 not found: ID does not exist"
Sep 11 15:57:36.988350 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988322    2559 scope.go:117] "RemoveContainer" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.988575 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988555    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"} err="failed to get container status \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": rpc error: code = NotFound desc = could not find container \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": container with ID starting with d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a not found: ID does not exist"
Sep 11 15:57:36.988617 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988575    2559 scope.go:117] "RemoveContainer" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.988806 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988788    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"} err="failed to get container status \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": rpc error: code = NotFound desc = could not find container \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": container with ID starting with 875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903 not found: ID does not exist"
Sep 11 15:57:36.988849 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.988806    2559 scope.go:117] "RemoveContainer" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.989045 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989027    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"} err="failed to get container status \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": rpc error: code = NotFound desc = could not find container \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": container with ID starting with 528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532 not found: ID does not exist"
Sep 11 15:57:36.989086 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989046    2559 scope.go:117] "RemoveContainer" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.989267 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989248    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"} err="failed to get container status \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": rpc error: code = NotFound desc = could not find container \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": container with ID starting with a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205 not found: ID does not exist"
Sep 11 15:57:36.989316 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989268    2559 scope.go:117] "RemoveContainer" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.989496 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989477    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"} err="failed to get container status \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": rpc error: code = NotFound desc = could not find container \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": container with ID starting with c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141 not found: ID does not exist"
Sep 11 15:57:36.989542 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989505    2559 scope.go:117] "RemoveContainer" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.989718 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989698    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"} err="failed to get container status \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": rpc error: code = NotFound desc = could not find container \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": container with ID starting with 24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd not found: ID does not exist"
Sep 11 15:57:36.989756 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989718    2559 scope.go:117] "RemoveContainer" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.989959 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989942    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"} err="failed to get container status \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": rpc error: code = NotFound desc = could not find container \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": container with ID starting with bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964 not found: ID does not exist"
Sep 11 15:57:36.990001 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.989959    2559 scope.go:117] "RemoveContainer" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.990165 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990149    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"} err="failed to get container status \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": rpc error: code = NotFound desc = could not find container \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": container with ID starting with d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a not found: ID does not exist"
Sep 11 15:57:36.990208 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990165    2559 scope.go:117] "RemoveContainer" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.990401 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990383    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"} err="failed to get container status \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": rpc error: code = NotFound desc = could not find container \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": container with ID starting with 875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903 not found: ID does not exist"
Sep 11 15:57:36.990401 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990402    2559 scope.go:117] "RemoveContainer" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.990611 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990590    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"} err="failed to get container status \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": rpc error: code = NotFound desc = could not find container \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": container with ID starting with 528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532 not found: ID does not exist"
Sep 11 15:57:36.990645 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990611    2559 scope.go:117] "RemoveContainer" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.990821 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990800    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"} err="failed to get container status \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": rpc error: code = NotFound desc = could not find container \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": container with ID starting with a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205 not found: ID does not exist"
Sep 11 15:57:36.990858 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.990822    2559 scope.go:117] "RemoveContainer" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.991054 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991037    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"} err="failed to get container status \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": rpc error: code = NotFound desc = could not find container \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": container with ID starting with c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141 not found: ID does not exist"
Sep 11 15:57:36.991089 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991055    2559 scope.go:117] "RemoveContainer" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.991285 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991269    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"} err="failed to get container status \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": rpc error: code = NotFound desc = could not find container \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": container with ID starting with 24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd not found: ID does not exist"
Sep 11 15:57:36.991330 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991286    2559 scope.go:117] "RemoveContainer" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.991503 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991486    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"} err="failed to get container status \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": rpc error: code = NotFound desc = could not find container \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": container with ID starting with bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964 not found: ID does not exist"
Sep 11 15:57:36.991552 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991504    2559 scope.go:117] "RemoveContainer" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.991729 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991712    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"} err="failed to get container status \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": rpc error: code = NotFound desc = could not find container \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": container with ID starting with d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a not found: ID does not exist"
Sep 11 15:57:36.991773 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991730    2559 scope.go:117] "RemoveContainer" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.991934 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991918    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"} err="failed to get container status \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": rpc error: code = NotFound desc = could not find container \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": container with ID starting with 875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903 not found: ID does not exist"
Sep 11 15:57:36.991987 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.991934    2559 scope.go:117] "RemoveContainer" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.992209 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.992188    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"} err="failed to get container status \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": rpc error: code = NotFound desc = could not find container \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": container with ID starting with 528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532 not found: ID does not exist"
Sep 11 15:57:36.992209 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.992208    2559 scope.go:117] "RemoveContainer" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.992666 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.992557    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"} err="failed to get container status \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": rpc error: code = NotFound desc = could not find container \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": container with ID starting with a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205 not found: ID does not exist"
Sep 11 15:57:36.992666 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.992583    2559 scope.go:117] "RemoveContainer" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.992893 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.992862    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"} err="failed to get container status \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": rpc error: code = NotFound desc = could not find container \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": container with ID starting with c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141 not found: ID does not exist"
Sep 11 15:57:36.992952 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.992895    2559 scope.go:117] "RemoveContainer" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.993149 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.993127    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"} err="failed to get container status \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": rpc error: code = NotFound desc = could not find container \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": container with ID starting with 24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd not found: ID does not exist"
Sep 11 15:57:36.993217 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.993151    2559 scope.go:117] "RemoveContainer" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.993411 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.993391    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"} err="failed to get container status \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": rpc error: code = NotFound desc = could not find container \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": container with ID starting with bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964 not found: ID does not exist"
Sep 11 15:57:36.993495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.993413    2559 scope.go:117] "RemoveContainer" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.993733 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.993715    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"} err="failed to get container status \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": rpc error: code = NotFound desc = could not find container \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": container with ID starting with d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a not found: ID does not exist"
Sep 11 15:57:36.993783 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.993735    2559 scope.go:117] "RemoveContainer" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.994009 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.993980    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"} err="failed to get container status \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": rpc error: code = NotFound desc = could not find container \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": container with ID starting with 875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903 not found: ID does not exist"
Sep 11 15:57:36.994009 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994010    2559 scope.go:117] "RemoveContainer" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.994279 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994252    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"} err="failed to get container status \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": rpc error: code = NotFound desc = could not find container \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": container with ID starting with 528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532 not found: ID does not exist"
Sep 11 15:57:36.994279 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994278    2559 scope.go:117] "RemoveContainer" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.994509 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994494    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"} err="failed to get container status \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": rpc error: code = NotFound desc = could not find container \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": container with ID starting with a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205 not found: ID does not exist"
Sep 11 15:57:36.994509 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994509    2559 scope.go:117] "RemoveContainer" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.994638 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994620    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:57:36.994693 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994678    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"} err="failed to get container status \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": rpc error: code = NotFound desc = could not find container \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": container with ID starting with c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141 not found: ID does not exist"
Sep 11 15:57:36.994732 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994693    2559 scope.go:117] "RemoveContainer" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.994885 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994863    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"} err="failed to get container status \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": rpc error: code = NotFound desc = could not find container \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": container with ID starting with 24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd not found: ID does not exist"
Sep 11 15:57:36.994885 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994882    2559 scope.go:117] "RemoveContainer" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994928    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="init-config-reloader"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994941    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="init-config-reloader"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994956    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-web"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994963    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-web"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994975    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="config-reloader"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994981    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="config-reloader"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994989    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="thanos-sidecar"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.994994    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="thanos-sidecar"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995007    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995012    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995019    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85831938-67dc-49ff-9d0f-9b62a3a6dc0d" containerName="registry"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995024    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="85831938-67dc-49ff-9d0f-9b62a3a6dc0d" containerName="registry"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995031    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-thanos"
Sep 11 15:57:36.995034 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995037    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-thanos"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995054    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="prometheus"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995062    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="prometheus"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995094    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"} err="failed to get container status \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": rpc error: code = NotFound desc = could not find container \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": container with ID starting with bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964 not found: ID does not exist"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995108    2559 scope.go:117] "RemoveContainer" containerID="d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995115    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-web"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995126    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy-thanos"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995135    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="85831938-67dc-49ff-9d0f-9b62a3a6dc0d" containerName="registry"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995145    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="kube-rbac-proxy"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995154    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="prometheus"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995161    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="config-reloader"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995169    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" containerName="thanos-sidecar"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995337    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a"} err="failed to get container status \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": rpc error: code = NotFound desc = could not find container \"d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a\": container with ID starting with d12f0b135e93d4134d043862e5ff9ffeaad6a0f48ce04da314fdcaa53a733d9a not found: ID does not exist"
Sep 11 15:57:36.995495 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995356    2559 scope.go:117] "RemoveContainer" containerID="875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"
Sep 11 15:57:36.995973 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995580    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903"} err="failed to get container status \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": rpc error: code = NotFound desc = could not find container \"875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903\": container with ID starting with 875cb9bf232e2c8d2220ed707ef69530cc26fc3c53537aaeeb6c1176d0212903 not found: ID does not exist"
Sep 11 15:57:36.995973 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995607    2559 scope.go:117] "RemoveContainer" containerID="528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"
Sep 11 15:57:36.995973 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995826    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532"} err="failed to get container status \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": rpc error: code = NotFound desc = could not find container \"528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532\": container with ID starting with 528be8400b0fc0e1d413941022cab2b3951e732b5b39e46b73b58b3fb78a1532 not found: ID does not exist"
Sep 11 15:57:36.995973 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.995840    2559 scope.go:117] "RemoveContainer" containerID="a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"
Sep 11 15:57:36.996107 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.996046    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205"} err="failed to get container status \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": rpc error: code = NotFound desc = could not find container \"a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205\": container with ID starting with a35abc3fe99bba6569288376859f192e3e2c2309b170eb7945ec5b01e9564205 not found: ID does not exist"
Sep 11 15:57:36.996107 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.996067    2559 scope.go:117] "RemoveContainer" containerID="c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"
Sep 11 15:57:36.996376 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.996356    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141"} err="failed to get container status \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": rpc error: code = NotFound desc = could not find container \"c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141\": container with ID starting with c5ef58e44ba504516b21f9a942605481881240cc9cec7b34ed98c06ba7e68141 not found: ID does not exist"
Sep 11 15:57:36.996430 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.996376    2559 scope.go:117] "RemoveContainer" containerID="24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"
Sep 11 15:57:36.996607 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.996591    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd"} err="failed to get container status \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": rpc error: code = NotFound desc = could not find container \"24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd\": container with ID starting with 24aa61e9e0935b536496c42e6429d0f17c8cd378820679bf8b5b7d075b8416fd not found: ID does not exist"
Sep 11 15:57:36.996645 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.996607    2559 scope.go:117] "RemoveContainer" containerID="bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"
Sep 11 15:57:36.996819 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:36.996802    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964"} err="failed to get container status \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": rpc error: code = NotFound desc = could not find container \"bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964\": container with ID starting with bc75473d937e76816931682c6a2c02e8ff6d694ba805063f619bb08c07282964 not found: ID does not exist"
Sep 11 15:57:37.000797 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.000778    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.007622 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.007596    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 15:57:37.008088 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008071    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 15:57:37.008512 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008494    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 15:57:37.008591 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008512    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 15:57:37.008591 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008499    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 15:57:37.008591 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008532    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 15:57:37.008591 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008543    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 15:57:37.008827 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008813    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 15:57:37.008886 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008831    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r78wg\""
Sep 11 15:57:37.008958 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008939    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 15:57:37.008999 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.008981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1brm7orfpcpsa\""
Sep 11 15:57:37.010967 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.010942    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 15:57:37.011103 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.011060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 15:57:37.012707 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.012465    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 15:57:37.012707 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.012625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 15:57:37.017003 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.016975    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 15:57:37.018667 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.018633    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:57:37.018667 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.018654    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 15:57:37.117499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117499 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117492    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117515    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117579    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117597    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-config\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117710 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmlqh\" (UniqueName: \"kubernetes.io/projected/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-kube-api-access-dmlqh\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.117996 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.117980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219223 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219223 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219223 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219223 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219557 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219557 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219557 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmlqh\" (UniqueName: \"kubernetes.io/projected/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-kube-api-access-dmlqh\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219557 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219557 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219798 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219798 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219798 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219798 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219798 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219798 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.219798 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219758    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-config\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.219907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.220086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.220524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.220793 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.220520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.221300 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.221225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.222981 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.222951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.223268 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.223185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.223543 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.223515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.223678 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.223530    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224211 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.223984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224211 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.224032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224211 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.224144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224505 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.224371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-config\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224505 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.224415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224505 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.224428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224619 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.224601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.224955 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.224933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.225311 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.225294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.225512 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.225492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.230196 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.230174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmlqh\" (UniqueName: \"kubernetes.io/projected/6dc46a2e-19c8-4f0e-a256-faaeac548d6e-kube-api-access-dmlqh\") pod \"prometheus-k8s-0\" (UID: \"6dc46a2e-19c8-4f0e-a256-faaeac548d6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.311123 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.311081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:57:37.599954 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.599851    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 15:57:37.602580 ip-10-0-129-137 kubenswrapper[2559]: W0911 15:57:37.602539    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc46a2e_19c8_4f0e_a256_faaeac548d6e.slice/crio-ada4c6a696aabb4d5527a2b3d82f52d3ee49cc80b19fc2b71126941e207e136c WatchSource:0}: Error finding container ada4c6a696aabb4d5527a2b3d82f52d3ee49cc80b19fc2b71126941e207e136c: Status 404 returned error can't find the container with id ada4c6a696aabb4d5527a2b3d82f52d3ee49cc80b19fc2b71126941e207e136c
Sep 11 15:57:37.937777 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.937681    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dc46a2e-19c8-4f0e-a256-faaeac548d6e" containerID="60f105b201d9b7c4372ee719b78721e089a7f758c28491ab90e835e885aebd53" exitCode=0
Sep 11 15:57:37.937951 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.937768    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerDied","Data":"60f105b201d9b7c4372ee719b78721e089a7f758c28491ab90e835e885aebd53"}
Sep 11 15:57:37.937951 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:37.937845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerStarted","Data":"ada4c6a696aabb4d5527a2b3d82f52d3ee49cc80b19fc2b71126941e207e136c"}
Sep 11 15:57:38.033974 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.033941    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd32269-f6ae-4c91-b417-1bc2b8eb4620" path="/var/lib/kubelet/pods/7bd32269-f6ae-4c91-b417-1bc2b8eb4620/volumes"
Sep 11 15:57:38.946825 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.946785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerStarted","Data":"6a0c1248c5f6024d8b68f094d1972b6c88e4a62ffe864ec928c1247ac652055a"}
Sep 11 15:57:38.947332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.946881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerStarted","Data":"1486a0c9fbf67f821cdf6ecf69c10a9172573d10cc45f6fe9f948700f38cd272"}
Sep 11 15:57:38.947332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.946907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerStarted","Data":"b73c05e0900b12ae897ffaa7b62fe2e1ffa6593bd480c3ed47d01a9007f9a3b9"}
Sep 11 15:57:38.947332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.946930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerStarted","Data":"60af082a993b2eccac069fde648d10ba18e4d3d1441aaaecaed4d228f24c2d42"}
Sep 11 15:57:38.947332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.946953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerStarted","Data":"b12e694dc17940ab0fbf87a77c4594791e78aa4b00a222cf18335e39d724c081"}
Sep 11 15:57:38.947332 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.946977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6dc46a2e-19c8-4f0e-a256-faaeac548d6e","Type":"ContainerStarted","Data":"d9b39ad820340788970e1f94b093920841ea5ab74784ee74e0bcb5bcda56b5b1"}
Sep 11 15:57:38.989032 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:38.988956    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9682000029999998 podStartE2EDuration="2.988933027s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.733024ms" isStatefulPod=false podCreationTimestamp="2026-09-11 15:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:57:38.985613338 +0000 UTC m=+155.409431031" watchObservedRunningTime="2026-09-11 15:57:38.988933027 +0000 UTC m=+155.412750723"
Sep 11 15:57:42.311366 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:57:42.311312    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:58:37.311622 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:58:37.311583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:58:37.327420 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:58:37.327391    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 15:58:38.191377 ip-10-0-129-137 kubenswrapper[2559]: I0911 15:58:38.191350    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:00:03.953603 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:03.953573    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:00:49.166694 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.166657    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"]
Sep 11 16:00:49.170039 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.170021    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.172849 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.172820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:00:49.172988 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.172895    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4fk8w\""
Sep 11 16:00:49.174019 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.174001    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 16:00:49.178664 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.178640    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"]
Sep 11 16:00:49.230125 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.230082    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87v2x\" (UniqueName: \"kubernetes.io/projected/49315e01-9b89-47c2-b6c1-d5e6a3cb9477-kube-api-access-87v2x\") pod \"openshift-lws-operator-fd8ccff4c-5fllx\" (UID: \"49315e01-9b89-47c2-b6c1-d5e6a3cb9477\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.230339 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.230133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49315e01-9b89-47c2-b6c1-d5e6a3cb9477-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5fllx\" (UID: \"49315e01-9b89-47c2-b6c1-d5e6a3cb9477\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.330965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.330916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87v2x\" (UniqueName: \"kubernetes.io/projected/49315e01-9b89-47c2-b6c1-d5e6a3cb9477-kube-api-access-87v2x\") pod \"openshift-lws-operator-fd8ccff4c-5fllx\" (UID: \"49315e01-9b89-47c2-b6c1-d5e6a3cb9477\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.330965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.330977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49315e01-9b89-47c2-b6c1-d5e6a3cb9477-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5fllx\" (UID: \"49315e01-9b89-47c2-b6c1-d5e6a3cb9477\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.331352 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.331335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49315e01-9b89-47c2-b6c1-d5e6a3cb9477-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5fllx\" (UID: \"49315e01-9b89-47c2-b6c1-d5e6a3cb9477\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.340303 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.340275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87v2x\" (UniqueName: \"kubernetes.io/projected/49315e01-9b89-47c2-b6c1-d5e6a3cb9477-kube-api-access-87v2x\") pod \"openshift-lws-operator-fd8ccff4c-5fllx\" (UID: \"49315e01-9b89-47c2-b6c1-d5e6a3cb9477\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.492693 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.492601    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"
Sep 11 16:00:49.765982 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.765953    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx"]
Sep 11 16:00:49.768772 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:00:49.768741    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49315e01_9b89_47c2_b6c1_d5e6a3cb9477.slice/crio-9e0700f5674146379ffa220b0345f0f60edbebd73ddb5debe077473eb966fef0 WatchSource:0}: Error finding container 9e0700f5674146379ffa220b0345f0f60edbebd73ddb5debe077473eb966fef0: Status 404 returned error can't find the container with id 9e0700f5674146379ffa220b0345f0f60edbebd73ddb5debe077473eb966fef0
Sep 11 16:00:49.770415 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:49.770386    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:00:50.688367 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:50.688326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx" event={"ID":"49315e01-9b89-47c2-b6c1-d5e6a3cb9477","Type":"ContainerStarted","Data":"9e0700f5674146379ffa220b0345f0f60edbebd73ddb5debe077473eb966fef0"}
Sep 11 16:00:53.701966 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:53.701927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx" event={"ID":"49315e01-9b89-47c2-b6c1-d5e6a3cb9477","Type":"ContainerStarted","Data":"c1d19d85d236b9375e86ffa84187919bea87f66fd2ac764ab70158191552ceee"}
Sep 11 16:00:53.718209 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:00:53.718162    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fllx" podStartSLOduration=1.707368853 podStartE2EDuration="4.718145794s" totalImagesPullingTime="3.010776941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:00:53.717655642 +0000 UTC m=+350.141473337" watchObservedRunningTime="2026-09-11 16:00:53.718145794 +0000 UTC m=+350.141963470"
Sep 11 16:01:11.001411 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.001321    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"]
Sep 11 16:01:11.007146 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.007121    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.013326 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.013300    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tm5qj\""
Sep 11 16:01:11.014988 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.014955    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 16:01:11.015103 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.015067    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 16:01:11.022710 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.022682    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 16:01:11.023436 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.023416    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"]
Sep 11 16:01:11.117204 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.117158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq9vq\" (UniqueName: \"kubernetes.io/projected/c343d096-a13f-4710-ba42-37b8e9a868f0-kube-api-access-pq9vq\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.117428 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.117220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c343d096-a13f-4710-ba42-37b8e9a868f0-metrics-cert\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.117428 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.117308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c343d096-a13f-4710-ba42-37b8e9a868f0-cert\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.117428 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.117333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c343d096-a13f-4710-ba42-37b8e9a868f0-manager-config\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.218003 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.217960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c343d096-a13f-4710-ba42-37b8e9a868f0-cert\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.218204 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.218010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c343d096-a13f-4710-ba42-37b8e9a868f0-manager-config\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.218204 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.218060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq9vq\" (UniqueName: \"kubernetes.io/projected/c343d096-a13f-4710-ba42-37b8e9a868f0-kube-api-access-pq9vq\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.218204 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.218117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c343d096-a13f-4710-ba42-37b8e9a868f0-metrics-cert\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.218811 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.218786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c343d096-a13f-4710-ba42-37b8e9a868f0-manager-config\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.220642 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.220620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c343d096-a13f-4710-ba42-37b8e9a868f0-cert\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.220714 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.220691    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c343d096-a13f-4710-ba42-37b8e9a868f0-metrics-cert\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.231964 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.231932    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq9vq\" (UniqueName: \"kubernetes.io/projected/c343d096-a13f-4710-ba42-37b8e9a868f0-kube-api-access-pq9vq\") pod \"lws-controller-manager-6848676c-xhm5j\" (UID: \"c343d096-a13f-4710-ba42-37b8e9a868f0\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.316723 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.316621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:11.605130 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.605052    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"]
Sep 11 16:01:11.608346 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:01:11.608315    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc343d096_a13f_4710_ba42_37b8e9a868f0.slice/crio-917cfce12d31cf777801c2d87028709ef5377b1179a5833ccaf644fbeaac321a WatchSource:0}: Error finding container 917cfce12d31cf777801c2d87028709ef5377b1179a5833ccaf644fbeaac321a: Status 404 returned error can't find the container with id 917cfce12d31cf777801c2d87028709ef5377b1179a5833ccaf644fbeaac321a
Sep 11 16:01:11.775094 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:11.775054    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j" event={"ID":"c343d096-a13f-4710-ba42-37b8e9a868f0","Type":"ContainerStarted","Data":"917cfce12d31cf777801c2d87028709ef5377b1179a5833ccaf644fbeaac321a"}
Sep 11 16:01:12.222649 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.222605    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"]
Sep 11 16:01:12.229600 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.229573    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.232574 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.232545    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 16:01:12.232707 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.232669    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wl6q7\""
Sep 11 16:01:12.233197 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.233178    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 16:01:12.233304 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.233179    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 16:01:12.233304 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.233296    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 16:01:12.269021 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.268986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"]
Sep 11 16:01:12.327804 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.327772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfc98944-c2b3-4087-bd88-f1f091c02f69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.327965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.327875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfc98944-c2b3-4087-bd88-f1f091c02f69-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.327965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.327922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfc98944-c2b3-4087-bd88-f1f091c02f69-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.327965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.327950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr895\" (UniqueName: \"kubernetes.io/projected/dfc98944-c2b3-4087-bd88-f1f091c02f69-kube-api-access-jr895\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.328083 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.327970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc98944-c2b3-4087-bd88-f1f091c02f69-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.428433 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.428392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfc98944-c2b3-4087-bd88-f1f091c02f69-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.428616 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.428443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfc98944-c2b3-4087-bd88-f1f091c02f69-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.428616 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.428465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr895\" (UniqueName: \"kubernetes.io/projected/dfc98944-c2b3-4087-bd88-f1f091c02f69-kube-api-access-jr895\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.428616 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.428486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc98944-c2b3-4087-bd88-f1f091c02f69-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.428782 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.428612    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfc98944-c2b3-4087-bd88-f1f091c02f69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.428874 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.428839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfc98944-c2b3-4087-bd88-f1f091c02f69-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.430863 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.430833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfc98944-c2b3-4087-bd88-f1f091c02f69-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.430988 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.430969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfc98944-c2b3-4087-bd88-f1f091c02f69-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.431261 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.431217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfc98944-c2b3-4087-bd88-f1f091c02f69-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.437318 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.437288    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr895\" (UniqueName: \"kubernetes.io/projected/dfc98944-c2b3-4087-bd88-f1f091c02f69-kube-api-access-jr895\") pod \"opendatahub-operator-controller-manager-58647b66f4-jq8r7\" (UID: \"dfc98944-c2b3-4087-bd88-f1f091c02f69\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.540205 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.540156    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:12.824400 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:12.824319    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"]
Sep 11 16:01:12.826617 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:01:12.826588    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc98944_c2b3_4087_bd88_f1f091c02f69.slice/crio-6dad07975ab2fc88cf78802cd93ab50cd83e51547f4e98d6869513f8f4a41543 WatchSource:0}: Error finding container 6dad07975ab2fc88cf78802cd93ab50cd83e51547f4e98d6869513f8f4a41543: Status 404 returned error can't find the container with id 6dad07975ab2fc88cf78802cd93ab50cd83e51547f4e98d6869513f8f4a41543
Sep 11 16:01:13.788087 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:13.787957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j" event={"ID":"c343d096-a13f-4710-ba42-37b8e9a868f0","Type":"ContainerStarted","Data":"1c0db842c21e405977764860dd5af8a3947f169a80ac0a2e5c72538878679149"}
Sep 11 16:01:13.788087 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:13.788049    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:13.790080 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:13.790050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7" event={"ID":"dfc98944-c2b3-4087-bd88-f1f091c02f69","Type":"ContainerStarted","Data":"6dad07975ab2fc88cf78802cd93ab50cd83e51547f4e98d6869513f8f4a41543"}
Sep 11 16:01:13.808736 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:13.808667    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j" podStartSLOduration=1.969400409 podStartE2EDuration="3.8086447s" totalImagesPullingTime="1.839244291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:13.805909671 +0000 UTC m=+370.229727385" watchObservedRunningTime="2026-09-11 16:01:13.8086447 +0000 UTC m=+370.232462375"
Sep 11 16:01:16.805086 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:16.805047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7" event={"ID":"dfc98944-c2b3-4087-bd88-f1f091c02f69","Type":"ContainerStarted","Data":"83a5a7e65f948ef949f66bd3cfa343ca982ddb7e7b76d51eb03a7952ac31e176"}
Sep 11 16:01:17.811275 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:17.811222    2559 generic.go:356] "Generic (PLEG): container finished" podID="dfc98944-c2b3-4087-bd88-f1f091c02f69" containerID="83a5a7e65f948ef949f66bd3cfa343ca982ddb7e7b76d51eb03a7952ac31e176" exitCode=0
Sep 11 16:01:17.811662 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:17.811310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7" event={"ID":"dfc98944-c2b3-4087-bd88-f1f091c02f69","Type":"ContainerDied","Data":"83a5a7e65f948ef949f66bd3cfa343ca982ddb7e7b76d51eb03a7952ac31e176"}
Sep 11 16:01:18.818062 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:18.818024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7" event={"ID":"dfc98944-c2b3-4087-bd88-f1f091c02f69","Type":"ContainerStarted","Data":"a585a6c769aa0d49c74184a3228b7dc449a0e178def1e4e76bbc51cbb3bf97ff"}
Sep 11 16:01:18.818469 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:18.818150    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:18.843218 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:18.843144    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7" podStartSLOduration=2.186889961 podStartE2EDuration="6.843126444s" totalImagesPullingTime="3.563678551s" totalInitContainerRuntime="1.092557932s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:18.841059478 +0000 UTC m=+375.264877190" watchObservedRunningTime="2026-09-11 16:01:18.843126444 +0000 UTC m=+375.266944138"
Sep 11 16:01:24.797368 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:24.797330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6848676c-xhm5j"
Sep 11 16:01:29.824772 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:29.824739    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-jq8r7"
Sep 11 16:01:32.301573 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.301531    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"]
Sep 11 16:01:32.304838 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.304819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.310098 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.310073    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 16:01:32.310812 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.310785    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-42btk\""
Sep 11 16:01:32.310927 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.310792    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 16:01:32.311409 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.311262    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 16:01:32.311409 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.311301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 16:01:32.322661 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.322626    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"]
Sep 11 16:01:32.402778 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.402732    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e2870b-bbbd-4422-bc75-23321a7efa18-tls-certs\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.402967 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.402803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lzrb\" (UniqueName: \"kubernetes.io/projected/c3e2870b-bbbd-4422-bc75-23321a7efa18-kube-api-access-9lzrb\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.402967 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.402824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3e2870b-bbbd-4422-bc75-23321a7efa18-tmp\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.503374 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.503339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lzrb\" (UniqueName: \"kubernetes.io/projected/c3e2870b-bbbd-4422-bc75-23321a7efa18-kube-api-access-9lzrb\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.503374 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.503377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3e2870b-bbbd-4422-bc75-23321a7efa18-tmp\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.503584 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.503429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e2870b-bbbd-4422-bc75-23321a7efa18-tls-certs\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.505835 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.505808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3e2870b-bbbd-4422-bc75-23321a7efa18-tmp\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.506047 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.506029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e2870b-bbbd-4422-bc75-23321a7efa18-tls-certs\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.522731 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.522699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lzrb\" (UniqueName: \"kubernetes.io/projected/c3e2870b-bbbd-4422-bc75-23321a7efa18-kube-api-access-9lzrb\") pod \"kube-auth-proxy-6c65944c75-52wr4\" (UID: \"c3e2870b-bbbd-4422-bc75-23321a7efa18\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.614769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.614674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"
Sep 11 16:01:32.892545 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:32.892516    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c65944c75-52wr4"]
Sep 11 16:01:32.895112 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:01:32.895080    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e2870b_bbbd_4422_bc75_23321a7efa18.slice/crio-425be65a9ecb8c37e6bbd0f66c4194fd047de5bc043728815bced39ac9d6d56a WatchSource:0}: Error finding container 425be65a9ecb8c37e6bbd0f66c4194fd047de5bc043728815bced39ac9d6d56a: Status 404 returned error can't find the container with id 425be65a9ecb8c37e6bbd0f66c4194fd047de5bc043728815bced39ac9d6d56a
Sep 11 16:01:33.886190 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:33.886150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4" event={"ID":"c3e2870b-bbbd-4422-bc75-23321a7efa18","Type":"ContainerStarted","Data":"425be65a9ecb8c37e6bbd0f66c4194fd047de5bc043728815bced39ac9d6d56a"}
Sep 11 16:01:35.896502 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:35.896466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4" event={"ID":"c3e2870b-bbbd-4422-bc75-23321a7efa18","Type":"ContainerStarted","Data":"d99e6d942d9d010f25e67e1d2b552acf28d32a1c6c19d707c1da1fe712a3bf09"}
Sep 11 16:01:35.920528 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:01:35.920459    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c65944c75-52wr4" podStartSLOduration=1.030827112 podStartE2EDuration="3.92044033s" totalImagesPullingTime="2.889613218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:35.918905706 +0000 UTC m=+392.342723401" watchObservedRunningTime="2026-09-11 16:01:35.92044033 +0000 UTC m=+392.344258027"
Sep 11 16:03:18.378286 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.378247    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"]
Sep 11 16:03:18.381106 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.381086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.384666 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.384639    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 16:03:18.384962 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.384916    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 16:03:18.385157 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.385139    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 16:03:18.385215 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.385148    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 16:03:18.385952 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.385938    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wrlk5\""
Sep 11 16:03:18.394077 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.394049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"]
Sep 11 16:03:18.489346 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.489315    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66b9839-f4ca-4b4b-aea6-08012f472837-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.489503 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.489375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a66b9839-f4ca-4b4b-aea6-08012f472837-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.489503 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.489433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5d5\" (UniqueName: \"kubernetes.io/projected/a66b9839-f4ca-4b4b-aea6-08012f472837-kube-api-access-gb5d5\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.590525 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.590478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a66b9839-f4ca-4b4b-aea6-08012f472837-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.590701 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.590566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5d5\" (UniqueName: \"kubernetes.io/projected/a66b9839-f4ca-4b4b-aea6-08012f472837-kube-api-access-gb5d5\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.590701 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.590596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66b9839-f4ca-4b4b-aea6-08012f472837-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.591173 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.591141    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a66b9839-f4ca-4b4b-aea6-08012f472837-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.592924 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.592906    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a66b9839-f4ca-4b4b-aea6-08012f472837-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.600157 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.600127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5d5\" (UniqueName: \"kubernetes.io/projected/a66b9839-f4ca-4b4b-aea6-08012f472837-kube-api-access-gb5d5\") pod \"kuadrant-console-plugin-fbdd4f445-tx82k\" (UID: \"a66b9839-f4ca-4b4b-aea6-08012f472837\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.690532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.690441    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"
Sep 11 16:03:18.965489 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:18.965421    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k"]
Sep 11 16:03:18.968138 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:03:18.968106    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda66b9839_f4ca_4b4b_aea6_08012f472837.slice/crio-f00f1f471959fb1afebfcf10667f802ef66710e5ee315f781f0e4e0dabe5f14b WatchSource:0}: Error finding container f00f1f471959fb1afebfcf10667f802ef66710e5ee315f781f0e4e0dabe5f14b: Status 404 returned error can't find the container with id f00f1f471959fb1afebfcf10667f802ef66710e5ee315f781f0e4e0dabe5f14b
Sep 11 16:03:19.341217 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:19.341181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k" event={"ID":"a66b9839-f4ca-4b4b-aea6-08012f472837","Type":"ContainerStarted","Data":"f00f1f471959fb1afebfcf10667f802ef66710e5ee315f781f0e4e0dabe5f14b"}
Sep 11 16:03:25.370994 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:25.370951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k" event={"ID":"a66b9839-f4ca-4b4b-aea6-08012f472837","Type":"ContainerStarted","Data":"ed9a3251bfc3876f50d1fc52dbb69b00bc14efb98d59b77cca00a42047998b31"}
Sep 11 16:03:25.390413 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:25.390356    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tx82k" podStartSLOduration=1.180192628 podStartE2EDuration="7.390339203s" totalImagesPullingTime="6.210146575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:25.388759859 +0000 UTC m=+501.812577553" watchObservedRunningTime="2026-09-11 16:03:25.390339203 +0000 UTC m=+501.814156897"
Sep 11 16:03:29.675368 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.675329    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"]
Sep 11 16:03:29.677786 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.677764    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:29.680743 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.680717    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xcpqd\""
Sep 11 16:03:29.711751 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.711720    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"]
Sep 11 16:03:29.792587 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.792548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88cb5487-4b3c-425d-b7bf-d86625bf14ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:29.792766 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.792601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcz6w\" (UniqueName: \"kubernetes.io/projected/88cb5487-4b3c-425d-b7bf-d86625bf14ea-kube-api-access-dcz6w\") pod \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:29.893884 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.893842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88cb5487-4b3c-425d-b7bf-d86625bf14ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:29.894049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.893896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcz6w\" (UniqueName: \"kubernetes.io/projected/88cb5487-4b3c-425d-b7bf-d86625bf14ea-kube-api-access-dcz6w\") pod \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:29.894270 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.894248    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88cb5487-4b3c-425d-b7bf-d86625bf14ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:29.904504 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.904471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcz6w\" (UniqueName: \"kubernetes.io/projected/88cb5487-4b3c-425d-b7bf-d86625bf14ea-kube-api-access-dcz6w\") pod \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:29.906278 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.906230    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"]
Sep 11 16:03:29.906544 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:29.906533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:30.184663 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.184635    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"]
Sep 11 16:03:30.186365 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:03:30.186336    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88cb5487_4b3c_425d_b7bf_d86625bf14ea.slice/crio-75b28b97bf4c154359e87b5e886ac70d48943da04b7ef9d78bcf7055bf9e7322 WatchSource:0}: Error finding container 75b28b97bf4c154359e87b5e886ac70d48943da04b7ef9d78bcf7055bf9e7322: Status 404 returned error can't find the container with id 75b28b97bf4c154359e87b5e886ac70d48943da04b7ef9d78bcf7055bf9e7322
Sep 11 16:03:30.383024 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.382980    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"]
Sep 11 16:03:30.389862 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.389830    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"]
Sep 11 16:03:30.473893 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.473860    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"]
Sep 11 16:03:30.476905 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.476882    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:30.503723 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.503687    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"]
Sep 11 16:03:30.599144 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.599099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlkt\" (UniqueName: \"kubernetes.io/projected/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-kube-api-access-9xlkt\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s9sxt\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:30.599144 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.599150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s9sxt\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:30.700702 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.700604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlkt\" (UniqueName: \"kubernetes.io/projected/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-kube-api-access-9xlkt\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s9sxt\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:30.700702 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.700649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s9sxt\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:30.701272 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.701153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s9sxt\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:30.714854 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.714825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlkt\" (UniqueName: \"kubernetes.io/projected/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-kube-api-access-9xlkt\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s9sxt\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:30.787949 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:30.787915    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:31.077691 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:31.077400    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"]
Sep 11 16:03:31.082059 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:03:31.082019    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd8e5e0_b293_4837_a6f9_f85f30a9df5b.slice/crio-20c7aa6cf9bb4a89b4dd0d3beb3f59473cb63dfbbc8d58ef91cff42175380b9b WatchSource:0}: Error finding container 20c7aa6cf9bb4a89b4dd0d3beb3f59473cb63dfbbc8d58ef91cff42175380b9b: Status 404 returned error can't find the container with id 20c7aa6cf9bb4a89b4dd0d3beb3f59473cb63dfbbc8d58ef91cff42175380b9b
Sep 11 16:03:31.405024 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:31.404930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt" event={"ID":"abd8e5e0-b293-4837-a6f9-f85f30a9df5b","Type":"ContainerStarted","Data":"20c7aa6cf9bb4a89b4dd0d3beb3f59473cb63dfbbc8d58ef91cff42175380b9b"}
Sep 11 16:03:34.423190 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.423142    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" containerName="manager" containerID="cri-o://e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9" gracePeriod=2
Sep 11 16:03:34.426502 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:03:34.426466    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:34.426864 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.426837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt" event={"ID":"abd8e5e0-b293-4837-a6f9-f85f30a9df5b","Type":"ContainerStarted","Data":"bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b"}
Sep 11 16:03:34.426977 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.426943    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:34.464028 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.463963    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt" podStartSLOduration=1.6065935850000002 podStartE2EDuration="4.463943243s" totalImagesPullingTime="2.857349658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:34.462619173 +0000 UTC m=+510.886436892" watchObservedRunningTime="2026-09-11 16:03:34.463943243 +0000 UTC m=+510.887760937"
Sep 11 16:03:34.465368 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:03:34.465337    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:34.648565 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.648537    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:34.651383 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:03:34.651358    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:34.740129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.740042    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcz6w\" (UniqueName: \"kubernetes.io/projected/88cb5487-4b3c-425d-b7bf-d86625bf14ea-kube-api-access-dcz6w\") pod \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") "
Sep 11 16:03:34.740129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.740102    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88cb5487-4b3c-425d-b7bf-d86625bf14ea-extensions-socket-volume\") pod \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\" (UID: \"88cb5487-4b3c-425d-b7bf-d86625bf14ea\") "
Sep 11 16:03:34.740425 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.740396    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cb5487-4b3c-425d-b7bf-d86625bf14ea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "88cb5487-4b3c-425d-b7bf-d86625bf14ea" (UID: "88cb5487-4b3c-425d-b7bf-d86625bf14ea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:03:34.742223 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.742200    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cb5487-4b3c-425d-b7bf-d86625bf14ea-kube-api-access-dcz6w" (OuterVolumeSpecName: "kube-api-access-dcz6w") pod "88cb5487-4b3c-425d-b7bf-d86625bf14ea" (UID: "88cb5487-4b3c-425d-b7bf-d86625bf14ea"). InnerVolumeSpecName "kube-api-access-dcz6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:03:34.841571 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.841536    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcz6w\" (UniqueName: \"kubernetes.io/projected/88cb5487-4b3c-425d-b7bf-d86625bf14ea-kube-api-access-dcz6w\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:03:34.841571 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:34.841565    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88cb5487-4b3c-425d-b7bf-d86625bf14ea-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:03:35.434953 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:35.434919    2559 generic.go:356] "Generic (PLEG): container finished" podID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" containerID="e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9" exitCode=2
Sep 11 16:03:35.435374 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:35.434966    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:35.435374 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:35.435022    2559 scope.go:117] "RemoveContainer" containerID="e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9"
Sep 11 16:03:35.438222 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:03:35.438198    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:35.444008 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:35.443986    2559 scope.go:117] "RemoveContainer" containerID="e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9"
Sep 11 16:03:35.444323 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:03:35.444296    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9\": container with ID starting with e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9 not found: ID does not exist" containerID="e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9"
Sep 11 16:03:35.444393 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:35.444335    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9"} err="failed to get container status \"e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9\": rpc error: code = NotFound desc = could not find container \"e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9\": container with ID starting with e04c1de24d5ee7450f02f6a4e0d6efbaf0da8d5ee40005954c259f6fa2bbfba9 not found: ID does not exist"
Sep 11 16:03:35.447705 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:03:35.447676    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-thf8q\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-thf8q"
Sep 11 16:03:36.034021 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:36.033987    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" path="/var/lib/kubelet/pods/88cb5487-4b3c-425d-b7bf-d86625bf14ea/volumes"
Sep 11 16:03:45.437708 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:45.437672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:57.835398 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:57.835299    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"]
Sep 11 16:03:57.835926 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:57.835558    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt" podUID="abd8e5e0-b293-4837-a6f9-f85f30a9df5b" containerName="manager" containerID="cri-o://bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b" gracePeriod=10
Sep 11 16:03:58.118455 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.118415    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:58.244713 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.244677    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xlkt\" (UniqueName: \"kubernetes.io/projected/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-kube-api-access-9xlkt\") pod \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") "
Sep 11 16:03:58.244880 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.244727    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-extensions-socket-volume\") pod \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\" (UID: \"abd8e5e0-b293-4837-a6f9-f85f30a9df5b\") "
Sep 11 16:03:58.245230 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.245202    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "abd8e5e0-b293-4837-a6f9-f85f30a9df5b" (UID: "abd8e5e0-b293-4837-a6f9-f85f30a9df5b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:03:58.246854 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.246828    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-kube-api-access-9xlkt" (OuterVolumeSpecName: "kube-api-access-9xlkt") pod "abd8e5e0-b293-4837-a6f9-f85f30a9df5b" (UID: "abd8e5e0-b293-4837-a6f9-f85f30a9df5b"). InnerVolumeSpecName "kube-api-access-9xlkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:03:58.345770 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.345670    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlkt\" (UniqueName: \"kubernetes.io/projected/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-kube-api-access-9xlkt\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:03:58.345770 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.345707    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abd8e5e0-b293-4837-a6f9-f85f30a9df5b-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:03:58.540362 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.540322    2559 generic.go:356] "Generic (PLEG): container finished" podID="abd8e5e0-b293-4837-a6f9-f85f30a9df5b" containerID="bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b" exitCode=0
Sep 11 16:03:58.540533 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.540377    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"
Sep 11 16:03:58.540533 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.540412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt" event={"ID":"abd8e5e0-b293-4837-a6f9-f85f30a9df5b","Type":"ContainerDied","Data":"bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b"}
Sep 11 16:03:58.540533 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.540468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt" event={"ID":"abd8e5e0-b293-4837-a6f9-f85f30a9df5b","Type":"ContainerDied","Data":"20c7aa6cf9bb4a89b4dd0d3beb3f59473cb63dfbbc8d58ef91cff42175380b9b"}
Sep 11 16:03:58.540533 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.540492    2559 scope.go:117] "RemoveContainer" containerID="bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b"
Sep 11 16:03:58.549530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.549501    2559 scope.go:117] "RemoveContainer" containerID="bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b"
Sep 11 16:03:58.549942 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:03:58.549905    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b\": container with ID starting with bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b not found: ID does not exist" containerID="bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b"
Sep 11 16:03:58.550053 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.549947    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b"} err="failed to get container status \"bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b\": rpc error: code = NotFound desc = could not find container \"bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b\": container with ID starting with bfd308d96198547bfd95c1150b246dd311667d06ba883577e9254b4593307f5b not found: ID does not exist"
Sep 11 16:03:58.579843 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.579807    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"]
Sep 11 16:03:58.591327 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:03:58.591294    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s9sxt"]
Sep 11 16:04:00.034915 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:00.034879    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd8e5e0-b293-4837-a6f9-f85f30a9df5b" path="/var/lib/kubelet/pods/abd8e5e0-b293-4837-a6f9-f85f30a9df5b/volumes"
Sep 11 16:04:17.344357 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.344320    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-phn2r"]
Sep 11 16:04:17.344744 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.344649    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd8e5e0-b293-4837-a6f9-f85f30a9df5b" containerName="manager"
Sep 11 16:04:17.344744 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.344660    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd8e5e0-b293-4837-a6f9-f85f30a9df5b" containerName="manager"
Sep 11 16:04:17.344744 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.344681    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" containerName="manager"
Sep 11 16:04:17.344744 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.344686    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" containerName="manager"
Sep 11 16:04:17.344744 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.344740    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="88cb5487-4b3c-425d-b7bf-d86625bf14ea" containerName="manager"
Sep 11 16:04:17.344903 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.344752    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd8e5e0-b293-4837-a6f9-f85f30a9df5b" containerName="manager"
Sep 11 16:04:17.347692 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.347667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.353569 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.353544    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 16:04:17.374958 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.374923    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-phn2r"]
Sep 11 16:04:17.404363 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.404312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5x8c\" (UniqueName: \"kubernetes.io/projected/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-kube-api-access-t5x8c\") pod \"limitador-limitador-8588749977-phn2r\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") " pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.404557 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.404382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-config-file\") pod \"limitador-limitador-8588749977-phn2r\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") " pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.441511 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.441464    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-phn2r"]
Sep 11 16:04:17.505309 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.505277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5x8c\" (UniqueName: \"kubernetes.io/projected/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-kube-api-access-t5x8c\") pod \"limitador-limitador-8588749977-phn2r\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") " pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.505309 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.505318    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-config-file\") pod \"limitador-limitador-8588749977-phn2r\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") " pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.505937 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.505918    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-config-file\") pod \"limitador-limitador-8588749977-phn2r\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") " pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.513869 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.513844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5x8c\" (UniqueName: \"kubernetes.io/projected/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-kube-api-access-t5x8c\") pod \"limitador-limitador-8588749977-phn2r\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") " pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.657702 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.657614    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:17.929747 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:17.929572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-phn2r"]
Sep 11 16:04:17.932631 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:04:17.932599    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82aaf080_1dd6_4642_b8b8_d10ec9df0bc1.slice/crio-6bf023b9047933c6d97003228015185dd5c87c960230a954523e534e252a3893 WatchSource:0}: Error finding container 6bf023b9047933c6d97003228015185dd5c87c960230a954523e534e252a3893: Status 404 returned error can't find the container with id 6bf023b9047933c6d97003228015185dd5c87c960230a954523e534e252a3893
Sep 11 16:04:18.244327 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.244231    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dmbkb"]
Sep 11 16:04:18.247060 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.247041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb"
Sep 11 16:04:18.255090 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.255064    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4469b\""
Sep 11 16:04:18.264962 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.264937    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dmbkb"]
Sep 11 16:04:18.312139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.312096    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpf97\" (UniqueName: \"kubernetes.io/projected/33f660a7-97ba-488f-b108-965c17b37551-kube-api-access-fpf97\") pod \"authorino-7bcfd8f9cd-dmbkb\" (UID: \"33f660a7-97ba-488f-b108-965c17b37551\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb"
Sep 11 16:04:18.413337 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.413297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpf97\" (UniqueName: \"kubernetes.io/projected/33f660a7-97ba-488f-b108-965c17b37551-kube-api-access-fpf97\") pod \"authorino-7bcfd8f9cd-dmbkb\" (UID: \"33f660a7-97ba-488f-b108-965c17b37551\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb"
Sep 11 16:04:18.424260 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.424209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpf97\" (UniqueName: \"kubernetes.io/projected/33f660a7-97ba-488f-b108-965c17b37551-kube-api-access-fpf97\") pod \"authorino-7bcfd8f9cd-dmbkb\" (UID: \"33f660a7-97ba-488f-b108-965c17b37551\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb"
Sep 11 16:04:18.556324 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.556231    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb"
Sep 11 16:04:18.632641 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.632599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-phn2r" event={"ID":"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1","Type":"ContainerStarted","Data":"6bf023b9047933c6d97003228015185dd5c87c960230a954523e534e252a3893"}
Sep 11 16:04:18.822626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:18.822572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dmbkb"]
Sep 11 16:04:18.825106 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:04:18.825078    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33f660a7_97ba_488f_b108_965c17b37551.slice/crio-8bd02aaace646fdb258dfe1934b43838a4995d8aac946dc869d3ad4a918a721c WatchSource:0}: Error finding container 8bd02aaace646fdb258dfe1934b43838a4995d8aac946dc869d3ad4a918a721c: Status 404 returned error can't find the container with id 8bd02aaace646fdb258dfe1934b43838a4995d8aac946dc869d3ad4a918a721c
Sep 11 16:04:19.640482 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:19.640447    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb" event={"ID":"33f660a7-97ba-488f-b108-965c17b37551","Type":"ContainerStarted","Data":"8bd02aaace646fdb258dfe1934b43838a4995d8aac946dc869d3ad4a918a721c"}
Sep 11 16:04:21.653516 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:21.653464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-phn2r" event={"ID":"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1","Type":"ContainerStarted","Data":"480b3b93055e02830c2ff8e0e451e84024f8f52da6b5fad4d4c43040918ea5f1"}
Sep 11 16:04:21.653987 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:21.653584    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:21.673499 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:21.673423    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-phn2r" podStartSLOduration=1.721247946 podStartE2EDuration="4.673402267s" totalImagesPullingTime="2.952154321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:04:21.670346904 +0000 UTC m=+558.094164600" watchObservedRunningTime="2026-09-11 16:04:21.673402267 +0000 UTC m=+558.097219963"
Sep 11 16:04:22.659764 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:22.659720    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb" event={"ID":"33f660a7-97ba-488f-b108-965c17b37551","Type":"ContainerStarted","Data":"035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556"}
Sep 11 16:04:22.677431 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:22.677375    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb" podStartSLOduration=1.732240996 podStartE2EDuration="4.677353946s" totalImagesPullingTime="2.94511295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:04:22.675903246 +0000 UTC m=+559.099720942" watchObservedRunningTime="2026-09-11 16:04:22.677353946 +0000 UTC m=+559.101171657"
Sep 11 16:04:25.291019 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.290979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dmbkb"]
Sep 11 16:04:25.291496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.291134    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb" podUID="33f660a7-97ba-488f-b108-965c17b37551" containerName="authorino" containerID="cri-o://035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556" gracePeriod=30
Sep 11 16:04:25.532922 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.532896    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb"
Sep 11 16:04:25.576754 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.576665    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpf97\" (UniqueName: \"kubernetes.io/projected/33f660a7-97ba-488f-b108-965c17b37551-kube-api-access-fpf97\") pod \"33f660a7-97ba-488f-b108-965c17b37551\" (UID: \"33f660a7-97ba-488f-b108-965c17b37551\") "
Sep 11 16:04:25.578749 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.578714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f660a7-97ba-488f-b108-965c17b37551-kube-api-access-fpf97" (OuterVolumeSpecName: "kube-api-access-fpf97") pod "33f660a7-97ba-488f-b108-965c17b37551" (UID: "33f660a7-97ba-488f-b108-965c17b37551"). InnerVolumeSpecName "kube-api-access-fpf97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:04:25.674438 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.674405    2559 generic.go:356] "Generic (PLEG): container finished" podID="33f660a7-97ba-488f-b108-965c17b37551" containerID="035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556" exitCode=0
Sep 11 16:04:25.674625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.674447    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb"
Sep 11 16:04:25.674625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.674481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb" event={"ID":"33f660a7-97ba-488f-b108-965c17b37551","Type":"ContainerDied","Data":"035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556"}
Sep 11 16:04:25.674625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.674514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dmbkb" event={"ID":"33f660a7-97ba-488f-b108-965c17b37551","Type":"ContainerDied","Data":"8bd02aaace646fdb258dfe1934b43838a4995d8aac946dc869d3ad4a918a721c"}
Sep 11 16:04:25.674625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.674531    2559 scope.go:117] "RemoveContainer" containerID="035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556"
Sep 11 16:04:25.677364 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.677342    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpf97\" (UniqueName: \"kubernetes.io/projected/33f660a7-97ba-488f-b108-965c17b37551-kube-api-access-fpf97\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:04:25.683424 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.683406    2559 scope.go:117] "RemoveContainer" containerID="035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556"
Sep 11 16:04:25.683709 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:04:25.683692    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556\": container with ID starting with 035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556 not found: ID does not exist" containerID="035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556"
Sep 11 16:04:25.683764 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.683716    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556"} err="failed to get container status \"035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556\": rpc error: code = NotFound desc = could not find container \"035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556\": container with ID starting with 035aa75e0800ad59343867a818e85b263c985b494f9aa4d175f45e7aebe96556 not found: ID does not exist"
Sep 11 16:04:25.699446 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.699415    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dmbkb"]
Sep 11 16:04:25.705773 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:25.705747    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dmbkb"]
Sep 11 16:04:26.036339 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:26.036306    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f660a7-97ba-488f-b108-965c17b37551" path="/var/lib/kubelet/pods/33f660a7-97ba-488f-b108-965c17b37551/volumes"
Sep 11 16:04:31.254303 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.254258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-phn2r"]
Sep 11 16:04:31.254898 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.254546    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-phn2r" podUID="82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" containerName="limitador" containerID="cri-o://480b3b93055e02830c2ff8e0e451e84024f8f52da6b5fad4d4c43040918ea5f1" gracePeriod=30
Sep 11 16:04:31.255281 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.255180    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:31.707642 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.707609    2559 generic.go:356] "Generic (PLEG): container finished" podID="82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" containerID="480b3b93055e02830c2ff8e0e451e84024f8f52da6b5fad4d4c43040918ea5f1" exitCode=0
Sep 11 16:04:31.707821 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.707691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-phn2r" event={"ID":"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1","Type":"ContainerDied","Data":"480b3b93055e02830c2ff8e0e451e84024f8f52da6b5fad4d4c43040918ea5f1"}
Sep 11 16:04:31.805044 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.805017    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:31.932053 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.931956    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5x8c\" (UniqueName: \"kubernetes.io/projected/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-kube-api-access-t5x8c\") pod \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") "
Sep 11 16:04:31.932053 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.932017    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-config-file\") pod \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\" (UID: \"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1\") "
Sep 11 16:04:31.932438 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.932411    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-config-file" (OuterVolumeSpecName: "config-file") pod "82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" (UID: "82aaf080-1dd6-4642-b8b8-d10ec9df0bc1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:04:31.934128 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:31.934105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-kube-api-access-t5x8c" (OuterVolumeSpecName: "kube-api-access-t5x8c") pod "82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" (UID: "82aaf080-1dd6-4642-b8b8-d10ec9df0bc1"). InnerVolumeSpecName "kube-api-access-t5x8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:04:32.032958 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:32.032924    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5x8c\" (UniqueName: \"kubernetes.io/projected/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-kube-api-access-t5x8c\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:04:32.032958 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:32.032952    2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1-config-file\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:04:32.713796 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:32.713769    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-phn2r"
Sep 11 16:04:32.714268 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:32.713803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-phn2r" event={"ID":"82aaf080-1dd6-4642-b8b8-d10ec9df0bc1","Type":"ContainerDied","Data":"6bf023b9047933c6d97003228015185dd5c87c960230a954523e534e252a3893"}
Sep 11 16:04:32.714268 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:32.713861    2559 scope.go:117] "RemoveContainer" containerID="480b3b93055e02830c2ff8e0e451e84024f8f52da6b5fad4d4c43040918ea5f1"
Sep 11 16:04:32.733717 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:32.733682    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-phn2r"]
Sep 11 16:04:32.738701 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:32.738660    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-phn2r"]
Sep 11 16:04:34.033569 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:34.033515    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" path="/var/lib/kubelet/pods/82aaf080-1dd6-4642-b8b8-d10ec9df0bc1/volumes"
Sep 11 16:04:37.078937 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.078902    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"]
Sep 11 16:04:37.079421 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.079223    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f660a7-97ba-488f-b108-965c17b37551" containerName="authorino"
Sep 11 16:04:37.079421 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.079248    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f660a7-97ba-488f-b108-965c17b37551" containerName="authorino"
Sep 11 16:04:37.079421 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.079258    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" containerName="limitador"
Sep 11 16:04:37.079421 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.079264    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" containerName="limitador"
Sep 11 16:04:37.079421 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.079333    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82aaf080-1dd6-4642-b8b8-d10ec9df0bc1" containerName="limitador"
Sep 11 16:04:37.079421 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.079342    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f660a7-97ba-488f-b108-965c17b37551" containerName="authorino"
Sep 11 16:04:37.082055 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.082032    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.090928 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.090905    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 16:04:37.091028 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.090912    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 16:04:37.092050 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.092031    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 16:04:37.092160 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.092035    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fw5b4\""
Sep 11 16:04:37.097086 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.097060    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"]
Sep 11 16:04:37.176303 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.176262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hb8p\" (UniqueName: \"kubernetes.io/projected/fcb06b3c-fdee-43a1-85b0-a9ede25197a0-kube-api-access-9hb8p\") pod \"postgres-69fb4594cf-7hprz\" (UID: \"fcb06b3c-fdee-43a1-85b0-a9ede25197a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.176303 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.176306    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06b3c-fdee-43a1-85b0-a9ede25197a0-data\") pod \"postgres-69fb4594cf-7hprz\" (UID: \"fcb06b3c-fdee-43a1-85b0-a9ede25197a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.277089 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.277049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hb8p\" (UniqueName: \"kubernetes.io/projected/fcb06b3c-fdee-43a1-85b0-a9ede25197a0-kube-api-access-9hb8p\") pod \"postgres-69fb4594cf-7hprz\" (UID: \"fcb06b3c-fdee-43a1-85b0-a9ede25197a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.277089 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.277094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06b3c-fdee-43a1-85b0-a9ede25197a0-data\") pod \"postgres-69fb4594cf-7hprz\" (UID: \"fcb06b3c-fdee-43a1-85b0-a9ede25197a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.277478 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.277463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06b3c-fdee-43a1-85b0-a9ede25197a0-data\") pod \"postgres-69fb4594cf-7hprz\" (UID: \"fcb06b3c-fdee-43a1-85b0-a9ede25197a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.286864 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.286840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hb8p\" (UniqueName: \"kubernetes.io/projected/fcb06b3c-fdee-43a1-85b0-a9ede25197a0-kube-api-access-9hb8p\") pod \"postgres-69fb4594cf-7hprz\" (UID: \"fcb06b3c-fdee-43a1-85b0-a9ede25197a0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.392012 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.391920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:37.667671 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.667643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"]
Sep 11 16:04:37.669627 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:04:37.669598    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb06b3c_fdee_43a1_85b0_a9ede25197a0.slice/crio-2d082c159ccf7278f9e28522c64b429ca662f1c3521c8a992acefef3f9a48b1f WatchSource:0}: Error finding container 2d082c159ccf7278f9e28522c64b429ca662f1c3521c8a992acefef3f9a48b1f: Status 404 returned error can't find the container with id 2d082c159ccf7278f9e28522c64b429ca662f1c3521c8a992acefef3f9a48b1f
Sep 11 16:04:37.742830 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:37.742790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz" event={"ID":"fcb06b3c-fdee-43a1-85b0-a9ede25197a0","Type":"ContainerStarted","Data":"2d082c159ccf7278f9e28522c64b429ca662f1c3521c8a992acefef3f9a48b1f"}
Sep 11 16:04:40.757699 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:40.757661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz" event={"ID":"fcb06b3c-fdee-43a1-85b0-a9ede25197a0","Type":"ContainerStarted","Data":"ab9c484811946b82ae5b4f051e40365e70851f7e7bb1cef7732d51edd6eee365"}
Sep 11 16:04:40.758193 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:40.757786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:40.774194 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:40.774137    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz" podStartSLOduration=1.243315315 podStartE2EDuration="3.774117544s" totalImagesPullingTime="2.530802229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:04:40.773375928 +0000 UTC m=+577.197193638" watchObservedRunningTime="2026-09-11 16:04:40.774117544 +0000 UTC m=+577.197935238"
Sep 11 16:04:46.791629 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:46.791595    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7hprz"
Sep 11 16:04:47.406251 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.406200    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-x249q"]
Sep 11 16:04:47.412220 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.412183    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x249q"
Sep 11 16:04:47.415477 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.415452    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4469b\""
Sep 11 16:04:47.418841 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.418813    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-x249q"]
Sep 11 16:04:47.528696 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.528658    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-x249q"]
Sep 11 16:04:47.528966 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:04:47.528945    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j5jb2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-x249q" podUID="19b83463-fd5a-431c-a213-e9a41158fb0e"
Sep 11 16:04:47.563064 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.563024    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sv6ws"]
Sep 11 16:04:47.565717 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.565693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jb2\" (UniqueName: \"kubernetes.io/projected/19b83463-fd5a-431c-a213-e9a41158fb0e-kube-api-access-j5jb2\") pod \"authorino-557f876f69-x249q\" (UID: \"19b83463-fd5a-431c-a213-e9a41158fb0e\") " pod="kuadrant-system/authorino-557f876f69-x249q"
Sep 11 16:04:47.567451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.567433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.570338 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.570316    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 16:04:47.576446 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.576411    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sv6ws"]
Sep 11 16:04:47.656246 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.656187    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sv6ws"]
Sep 11 16:04:47.656610 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:04:47.656548    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p2mxj tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-sv6ws" podUID="9cb00161-24cb-440f-af3c-64b7eed134c0"
Sep 11 16:04:47.666712 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.666675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mxj\" (UniqueName: \"kubernetes.io/projected/9cb00161-24cb-440f-af3c-64b7eed134c0-kube-api-access-p2mxj\") pod \"authorino-6d4dfc649f-sv6ws\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") " pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.666803 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.666754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jb2\" (UniqueName: \"kubernetes.io/projected/19b83463-fd5a-431c-a213-e9a41158fb0e-kube-api-access-j5jb2\") pod \"authorino-557f876f69-x249q\" (UID: \"19b83463-fd5a-431c-a213-e9a41158fb0e\") " pod="kuadrant-system/authorino-557f876f69-x249q"
Sep 11 16:04:47.666840 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.666812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb00161-24cb-440f-af3c-64b7eed134c0-tls-cert\") pod \"authorino-6d4dfc649f-sv6ws\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") " pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.690817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.690781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jb2\" (UniqueName: \"kubernetes.io/projected/19b83463-fd5a-431c-a213-e9a41158fb0e-kube-api-access-j5jb2\") pod \"authorino-557f876f69-x249q\" (UID: \"19b83463-fd5a-431c-a213-e9a41158fb0e\") " pod="kuadrant-system/authorino-557f876f69-x249q"
Sep 11 16:04:47.696079 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.696045    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58d858cfdc-fqvb2"]
Sep 11 16:04:47.700917 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.700892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:47.709256 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.709213    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58d858cfdc-fqvb2"]
Sep 11 16:04:47.767599 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.767561    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb00161-24cb-440f-af3c-64b7eed134c0-tls-cert\") pod \"authorino-6d4dfc649f-sv6ws\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") " pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.767599 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.767609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mxj\" (UniqueName: \"kubernetes.io/projected/9cb00161-24cb-440f-af3c-64b7eed134c0-kube-api-access-p2mxj\") pod \"authorino-6d4dfc649f-sv6ws\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") " pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.770141 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.770118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb00161-24cb-440f-af3c-64b7eed134c0-tls-cert\") pod \"authorino-6d4dfc649f-sv6ws\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") " pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.776577 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.776533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mxj\" (UniqueName: \"kubernetes.io/projected/9cb00161-24cb-440f-af3c-64b7eed134c0-kube-api-access-p2mxj\") pod \"authorino-6d4dfc649f-sv6ws\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") " pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.789853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.789818    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.790047 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.789818    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x249q"
Sep 11 16:04:47.798537 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.798510    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:47.802130 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.802100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x249q"
Sep 11 16:04:47.869179 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.869125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lb5l\" (UniqueName: \"kubernetes.io/projected/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-kube-api-access-8lb5l\") pod \"authorino-58d858cfdc-fqvb2\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") " pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:47.869397 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.869292    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-tls-cert\") pod \"authorino-58d858cfdc-fqvb2\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") " pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:47.970720 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.970616    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5jb2\" (UniqueName: \"kubernetes.io/projected/19b83463-fd5a-431c-a213-e9a41158fb0e-kube-api-access-j5jb2\") pod \"19b83463-fd5a-431c-a213-e9a41158fb0e\" (UID: \"19b83463-fd5a-431c-a213-e9a41158fb0e\") "
Sep 11 16:04:47.970720 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.970708    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb00161-24cb-440f-af3c-64b7eed134c0-tls-cert\") pod \"9cb00161-24cb-440f-af3c-64b7eed134c0\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") "
Sep 11 16:04:47.970948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.970745    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2mxj\" (UniqueName: \"kubernetes.io/projected/9cb00161-24cb-440f-af3c-64b7eed134c0-kube-api-access-p2mxj\") pod \"9cb00161-24cb-440f-af3c-64b7eed134c0\" (UID: \"9cb00161-24cb-440f-af3c-64b7eed134c0\") "
Sep 11 16:04:47.971008 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.970953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-tls-cert\") pod \"authorino-58d858cfdc-fqvb2\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") " pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:47.971067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.971010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lb5l\" (UniqueName: \"kubernetes.io/projected/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-kube-api-access-8lb5l\") pod \"authorino-58d858cfdc-fqvb2\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") " pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:47.972948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.972917    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb00161-24cb-440f-af3c-64b7eed134c0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9cb00161-24cb-440f-af3c-64b7eed134c0" (UID: "9cb00161-24cb-440f-af3c-64b7eed134c0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:04:47.973079 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.973061    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb00161-24cb-440f-af3c-64b7eed134c0-kube-api-access-p2mxj" (OuterVolumeSpecName: "kube-api-access-p2mxj") pod "9cb00161-24cb-440f-af3c-64b7eed134c0" (UID: "9cb00161-24cb-440f-af3c-64b7eed134c0"). InnerVolumeSpecName "kube-api-access-p2mxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:04:47.973306 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.973281    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b83463-fd5a-431c-a213-e9a41158fb0e-kube-api-access-j5jb2" (OuterVolumeSpecName: "kube-api-access-j5jb2") pod "19b83463-fd5a-431c-a213-e9a41158fb0e" (UID: "19b83463-fd5a-431c-a213-e9a41158fb0e"). InnerVolumeSpecName "kube-api-access-j5jb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:04:47.973681 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.973662    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-tls-cert\") pod \"authorino-58d858cfdc-fqvb2\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") " pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:47.980262 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:47.980214    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lb5l\" (UniqueName: \"kubernetes.io/projected/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-kube-api-access-8lb5l\") pod \"authorino-58d858cfdc-fqvb2\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") " pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:48.011543 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.011505    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:04:48.072866 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.072115    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb00161-24cb-440f-af3c-64b7eed134c0-tls-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:04:48.072866 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.072162    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mxj\" (UniqueName: \"kubernetes.io/projected/9cb00161-24cb-440f-af3c-64b7eed134c0-kube-api-access-p2mxj\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:04:48.072866 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.072177    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jb2\" (UniqueName: \"kubernetes.io/projected/19b83463-fd5a-431c-a213-e9a41158fb0e-kube-api-access-j5jb2\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:04:48.285795 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.285766    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58d858cfdc-fqvb2"]
Sep 11 16:04:48.287606 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:04:48.287577    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf60c0b8e_8e04_41a0_b48e_ddfb290b7c7f.slice/crio-d7f69e5aeb34878c25b321e882f393de9ef1772890bf15fb3bd7496a324a734e WatchSource:0}: Error finding container d7f69e5aeb34878c25b321e882f393de9ef1772890bf15fb3bd7496a324a734e: Status 404 returned error can't find the container with id d7f69e5aeb34878c25b321e882f393de9ef1772890bf15fb3bd7496a324a734e
Sep 11 16:04:48.795648 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.795615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d858cfdc-fqvb2" event={"ID":"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f","Type":"ContainerStarted","Data":"d7f69e5aeb34878c25b321e882f393de9ef1772890bf15fb3bd7496a324a734e"}
Sep 11 16:04:48.795648 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.795648    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x249q"
Sep 11 16:04:48.795921 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.795812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sv6ws"
Sep 11 16:04:48.829382 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.829345    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-x249q"]
Sep 11 16:04:48.831276 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.831230    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-x249q"]
Sep 11 16:04:48.854406 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.854373    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sv6ws"]
Sep 11 16:04:48.859350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:48.859315    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sv6ws"]
Sep 11 16:04:50.034328 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:50.034288    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19b83463-fd5a-431c-a213-e9a41158fb0e" path="/var/lib/kubelet/pods/19b83463-fd5a-431c-a213-e9a41158fb0e/volumes"
Sep 11 16:04:50.034839 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:50.034719    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cb00161-24cb-440f-af3c-64b7eed134c0" path="/var/lib/kubelet/pods/9cb00161-24cb-440f-af3c-64b7eed134c0/volumes"
Sep 11 16:04:51.811350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:51.811310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d858cfdc-fqvb2" event={"ID":"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f","Type":"ContainerStarted","Data":"c05160206c7ca3ad62851f320b4c9f455b13daf68058e96d2613edd7421c9173"}
Sep 11 16:04:51.830682 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:04:51.830616    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58d858cfdc-fqvb2" podStartSLOduration=1.7836955030000001 podStartE2EDuration="4.830598654s" totalImagesPullingTime="3.046903151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:04:51.829583485 +0000 UTC m=+588.253401179" watchObservedRunningTime="2026-09-11 16:04:51.830598654 +0000 UTC m=+588.254416392"
Sep 11 16:05:35.141839 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.141754    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"]
Sep 11 16:05:35.145071 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.145049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.151873 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.151850    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 16:05:35.152080 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.152058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7mtgg\""
Sep 11 16:05:35.159877 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.159845    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"]
Sep 11 16:05:35.267956 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.267911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2720e4-65c1-4a80-8b03-5bc595a9a8d2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-g5d4s\" (UID: \"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.268164 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.267991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vkx\" (UniqueName: \"kubernetes.io/projected/7f2720e4-65c1-4a80-8b03-5bc595a9a8d2-kube-api-access-88vkx\") pod \"payload-pre-processing-59fbc47bf8-g5d4s\" (UID: \"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.369219 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.369166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2720e4-65c1-4a80-8b03-5bc595a9a8d2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-g5d4s\" (UID: \"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.369463 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.369288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vkx\" (UniqueName: \"kubernetes.io/projected/7f2720e4-65c1-4a80-8b03-5bc595a9a8d2-kube-api-access-88vkx\") pod \"payload-pre-processing-59fbc47bf8-g5d4s\" (UID: \"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.370152 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.370113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2720e4-65c1-4a80-8b03-5bc595a9a8d2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-g5d4s\" (UID: \"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.383579 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.383540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vkx\" (UniqueName: \"kubernetes.io/projected/7f2720e4-65c1-4a80-8b03-5bc595a9a8d2-kube-api-access-88vkx\") pod \"payload-pre-processing-59fbc47bf8-g5d4s\" (UID: \"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.455407 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.455314    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:35.744901 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:35.744870    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"]
Sep 11 16:05:35.745701 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:05:35.745674    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2720e4_65c1_4a80_8b03_5bc595a9a8d2.slice/crio-16de93e6b6eb456b1432ace13f154f03188c795fec9299bfdf5e4c06e7788f18 WatchSource:0}: Error finding container 16de93e6b6eb456b1432ace13f154f03188c795fec9299bfdf5e4c06e7788f18: Status 404 returned error can't find the container with id 16de93e6b6eb456b1432ace13f154f03188c795fec9299bfdf5e4c06e7788f18
Sep 11 16:05:36.015576 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:36.015538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s" event={"ID":"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2","Type":"ContainerStarted","Data":"16de93e6b6eb456b1432ace13f154f03188c795fec9299bfdf5e4c06e7788f18"}
Sep 11 16:05:39.031795 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:39.031758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s" event={"ID":"7f2720e4-65c1-4a80-8b03-5bc595a9a8d2","Type":"ContainerStarted","Data":"7759fd9a62239992d55ddcc424f648b3720554738466981cf4ac35f3e2d951a3"}
Sep 11 16:05:39.032302 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:39.031821    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:50.037983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:50.037947    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s"
Sep 11 16:05:50.056595 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:50.056542    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-g5d4s" podStartSLOduration=12.284901098 podStartE2EDuration="15.056525421s" totalImagesPullingTime="2.771624323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:05:39.059583184 +0000 UTC m=+635.483400878" watchObservedRunningTime="2026-09-11 16:05:50.056525421 +0000 UTC m=+646.480343115"
Sep 11 16:05:58.530938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.530894    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"]
Sep 11 16:05:58.534176 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.534153    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.539588 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.539564    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sxmds\""
Sep 11 16:05:58.539723 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.539567    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 16:05:58.540489 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.540459    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 16:05:58.540626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.540508    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 16:05:58.555128 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.555098    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"]
Sep 11 16:05:58.581737 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.581700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t5xj\" (UniqueName: \"kubernetes.io/projected/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-kube-api-access-7t5xj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.581737 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.581738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.581948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.581770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.581948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.581863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.581948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.581913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.581948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.581938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.682716 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.682675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t5xj\" (UniqueName: \"kubernetes.io/projected/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-kube-api-access-7t5xj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.682716 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.682718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.682974 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.682755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.682974 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.682791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.682974 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.682826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.682974 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.682846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.683337 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.683308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.683425 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.683319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.683425 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.683368    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.685129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.685108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.685321 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.685305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.694974 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.694938    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t5xj\" (UniqueName: \"kubernetes.io/projected/f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d-kube-api-access-7t5xj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw\" (UID: \"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:58.843997 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:58.843904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:05:59.130591 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:59.130562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"]
Sep 11 16:05:59.133126 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:05:59.133095    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e68c94_3dbf_4b23_8733_7e8d83cd6a7d.slice/crio-bdd043b7b95b186c7971d4cfdd506a5d27ae7c3b1df7a85436a288e3cbb6ab27 WatchSource:0}: Error finding container bdd043b7b95b186c7971d4cfdd506a5d27ae7c3b1df7a85436a288e3cbb6ab27: Status 404 returned error can't find the container with id bdd043b7b95b186c7971d4cfdd506a5d27ae7c3b1df7a85436a288e3cbb6ab27
Sep 11 16:05:59.134589 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:05:59.134573    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:06:00.124800 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:00.124761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw" event={"ID":"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d","Type":"ContainerStarted","Data":"bdd043b7b95b186c7971d4cfdd506a5d27ae7c3b1df7a85436a288e3cbb6ab27"}
Sep 11 16:06:01.131442 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:01.131409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw" event={"ID":"f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d","Type":"ContainerStarted","Data":"1ecac550b9f270df11180a1c8e00250ffc2fc9d8fbca441e3a4e866ab3ccc8bf"}
Sep 11 16:06:01.151528 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:01.151475    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw" podStartSLOduration=1.756611913 podStartE2EDuration="3.151455809s" totalImagesPullingTime="1.394843896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:06:01.150041756 +0000 UTC m=+657.573859474" watchObservedRunningTime="2026-09-11 16:06:01.151455809 +0000 UTC m=+657.575273502"
Sep 11 16:06:08.844343 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:08.844305    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:06:08.844784 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:08.844367    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:06:08.857289 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:08.857259    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:06:09.178078 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:09.177977    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw"
Sep 11 16:06:29.022957 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.022922    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"]
Sep 11 16:06:29.026173 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.026155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.030547 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.030526    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 16:06:29.046867 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.046839    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"]
Sep 11 16:06:29.153889 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.153842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7606f832-d8dd-40ce-9451-50bf9ce12c9c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.154089 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.153949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.154089 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.153991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.154089 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.154032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.154266 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.154107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4ch\" (UniqueName: \"kubernetes.io/projected/7606f832-d8dd-40ce-9451-50bf9ce12c9c-kube-api-access-wh4ch\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.154266 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.154179    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255047 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255271 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4ch\" (UniqueName: \"kubernetes.io/projected/7606f832-d8dd-40ce-9451-50bf9ce12c9c-kube-api-access-wh4ch\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255271 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255271 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7606f832-d8dd-40ce-9451-50bf9ce12c9c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255271 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255271 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255571 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255571 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.255687 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.255661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.257892 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.257857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7606f832-d8dd-40ce-9451-50bf9ce12c9c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.258348 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.258319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7606f832-d8dd-40ce-9451-50bf9ce12c9c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.267331 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.267304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4ch\" (UniqueName: \"kubernetes.io/projected/7606f832-d8dd-40ce-9451-50bf9ce12c9c-kube-api-access-wh4ch\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q\" (UID: \"7606f832-d8dd-40ce-9451-50bf9ce12c9c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.337294 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.337177    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:29.617643 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:29.617616    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"]
Sep 11 16:06:29.620348 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:06:29.620316    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7606f832_d8dd_40ce_9451_50bf9ce12c9c.slice/crio-7da5d8078846742736276818e26ddb33eb6725046c0ac4ea3a54ec91496f8fc3 WatchSource:0}: Error finding container 7da5d8078846742736276818e26ddb33eb6725046c0ac4ea3a54ec91496f8fc3: Status 404 returned error can't find the container with id 7da5d8078846742736276818e26ddb33eb6725046c0ac4ea3a54ec91496f8fc3
Sep 11 16:06:30.267103 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:30.267068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q" event={"ID":"7606f832-d8dd-40ce-9451-50bf9ce12c9c","Type":"ContainerStarted","Data":"2efa49ea54c47fb92be5d3b60cf567fe8b155581944d46415d4026531e1da5f5"}
Sep 11 16:06:30.267466 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:30.267159    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q" event={"ID":"7606f832-d8dd-40ce-9451-50bf9ce12c9c","Type":"ContainerStarted","Data":"7da5d8078846742736276818e26ddb33eb6725046c0ac4ea3a54ec91496f8fc3"}
Sep 11 16:06:30.294497 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:30.294424    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q" podStartSLOduration=1.102394536 podStartE2EDuration="1.294401426s" totalImagesPullingTime="192.00689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:06:30.290317531 +0000 UTC m=+686.714135227" watchObservedRunningTime="2026-09-11 16:06:30.294401426 +0000 UTC m=+686.718219121"
Sep 11 16:06:39.337544 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:39.337497    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:39.338159 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:39.337578    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:39.350257 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:39.350209    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:06:40.324627 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:06:40.324593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q"
Sep 11 16:07:04.603097 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.603008    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"]
Sep 11 16:07:04.605511 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.605493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.608827 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.608798    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 16:07:04.618398 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.618369    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"]
Sep 11 16:07:04.681960 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.681922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xgn\" (UniqueName: \"kubernetes.io/projected/73f35b9e-d767-420e-98b0-9f2102a05865-kube-api-access-46xgn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.682156 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.681969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.682156 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.682047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.682156 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.682082    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.682343 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.682159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.682343 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.682218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f35b9e-d767-420e-98b0-9f2102a05865-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783434 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xgn\" (UniqueName: \"kubernetes.io/projected/73f35b9e-d767-420e-98b0-9f2102a05865-kube-api-access-46xgn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783434 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783688 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783688 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783688 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783876 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f35b9e-d767-420e-98b0-9f2102a05865-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783876 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.783954 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.784007 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.783985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.785943 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.785911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f35b9e-d767-420e-98b0-9f2102a05865-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.786083 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.786066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f35b9e-d767-420e-98b0-9f2102a05865-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.794196 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.794171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xgn\" (UniqueName: \"kubernetes.io/projected/73f35b9e-d767-420e-98b0-9f2102a05865-kube-api-access-46xgn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt\" (UID: \"73f35b9e-d767-420e-98b0-9f2102a05865\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:04.917154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:04.917054    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:05.190401 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:05.190375    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"]
Sep 11 16:07:05.192081 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:07:05.192059    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f35b9e_d767_420e_98b0_9f2102a05865.slice/crio-fb4eeb7fe6d6b39398e2864a75da2bdc04dd4b53d5673bf03effd608dd0687f1 WatchSource:0}: Error finding container fb4eeb7fe6d6b39398e2864a75da2bdc04dd4b53d5673bf03effd608dd0687f1: Status 404 returned error can't find the container with id fb4eeb7fe6d6b39398e2864a75da2bdc04dd4b53d5673bf03effd608dd0687f1
Sep 11 16:07:05.439886 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:05.439829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt" event={"ID":"73f35b9e-d767-420e-98b0-9f2102a05865","Type":"ContainerStarted","Data":"fb4eeb7fe6d6b39398e2864a75da2bdc04dd4b53d5673bf03effd608dd0687f1"}
Sep 11 16:07:06.447299 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:06.447261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt" event={"ID":"73f35b9e-d767-420e-98b0-9f2102a05865","Type":"ContainerStarted","Data":"17ba02f53fb54773e0b565790c93d41f01d72b3648c33ec5934a7d5dffceaba2"}
Sep 11 16:07:06.470069 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:06.470010    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt" podStartSLOduration=2.278525904 podStartE2EDuration="2.469971585s" totalImagesPullingTime="191.445681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:07:06.468805965 +0000 UTC m=+722.892623679" watchObservedRunningTime="2026-09-11 16:07:06.469971585 +0000 UTC m=+722.893789278"
Sep 11 16:07:14.917900 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:14.917860    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:14.918414 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:14.917921    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:14.930826 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:14.930798    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:07:15.499012 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:07:15.498981    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt"
Sep 11 16:09:15.912069 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:15.912034    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58d858cfdc-fqvb2"]
Sep 11 16:09:15.912726 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:15.912223    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58d858cfdc-fqvb2" podUID="f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" containerName="authorino" containerID="cri-o://c05160206c7ca3ad62851f320b4c9f455b13daf68058e96d2613edd7421c9173" gracePeriod=30
Sep 11 16:09:16.056678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.056641    2559 generic.go:356] "Generic (PLEG): container finished" podID="f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" containerID="c05160206c7ca3ad62851f320b4c9f455b13daf68058e96d2613edd7421c9173" exitCode=0
Sep 11 16:09:16.056846 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.056719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d858cfdc-fqvb2" event={"ID":"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f","Type":"ContainerDied","Data":"c05160206c7ca3ad62851f320b4c9f455b13daf68058e96d2613edd7421c9173"}
Sep 11 16:09:16.167346 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.167282    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:09:16.286092 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.286055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-tls-cert\") pod \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") "
Sep 11 16:09:16.286281 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.286184    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lb5l\" (UniqueName: \"kubernetes.io/projected/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-kube-api-access-8lb5l\") pod \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\" (UID: \"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f\") "
Sep 11 16:09:16.288147 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.288109    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-kube-api-access-8lb5l" (OuterVolumeSpecName: "kube-api-access-8lb5l") pod "f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" (UID: "f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f"). InnerVolumeSpecName "kube-api-access-8lb5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:09:16.296082 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.296054    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" (UID: "f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:09:16.387539 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.387499    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lb5l\" (UniqueName: \"kubernetes.io/projected/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-kube-api-access-8lb5l\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:09:16.387539 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:16.387532    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f-tls-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:09:17.062683 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:17.062644    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d858cfdc-fqvb2" event={"ID":"f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f","Type":"ContainerDied","Data":"d7f69e5aeb34878c25b321e882f393de9ef1772890bf15fb3bd7496a324a734e"}
Sep 11 16:09:17.063074 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:17.062696    2559 scope.go:117] "RemoveContainer" containerID="c05160206c7ca3ad62851f320b4c9f455b13daf68058e96d2613edd7421c9173"
Sep 11 16:09:17.063074 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:17.062664    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d858cfdc-fqvb2"
Sep 11 16:09:17.108079 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:17.108047    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58d858cfdc-fqvb2"]
Sep 11 16:09:17.124769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:17.124736    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58d858cfdc-fqvb2"]
Sep 11 16:09:18.033503 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:18.033468    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" path="/var/lib/kubelet/pods/f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f/volumes"
Sep 11 16:09:38.906940 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.906901    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"]
Sep 11 16:09:38.907336 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.907286    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" containerName="authorino"
Sep 11 16:09:38.907336 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.907299    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" containerName="authorino"
Sep 11 16:09:38.907415 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.907364    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60c0b8e-8e04-41a0-b48e-ddfb290b7c7f" containerName="authorino"
Sep 11 16:09:38.910177 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.910161    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:38.913746 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.913720    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8e3121\""
Sep 11 16:09:38.914368 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.914348    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8e3121-dockercfg-2bpbk\""
Sep 11 16:09:38.926485 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.926457    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"]
Sep 11 16:09:38.960306 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.960270    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"]
Sep 11 16:09:38.963927 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.963909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:38.978555 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.978521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"]
Sep 11 16:09:38.984945 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.984913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eed963-6f72-4a6f-b7c7-ddb0529d0844-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:38.985094 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.985024    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a800ac6-982c-486f-9c47-e57a2a34770b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:38.985094 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.985048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckt6g\" (UniqueName: \"kubernetes.io/projected/5a800ac6-982c-486f-9c47-e57a2a34770b-kube-api-access-ckt6g\") pod \"payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:38.985220 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:38.985103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6w54\" (UniqueName: \"kubernetes.io/projected/88eed963-6f72-4a6f-b7c7-ddb0529d0844-kube-api-access-j6w54\") pod \"payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:39.085838 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.085801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6w54\" (UniqueName: \"kubernetes.io/projected/88eed963-6f72-4a6f-b7c7-ddb0529d0844-kube-api-access-j6w54\") pod \"payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:39.086056 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.085881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eed963-6f72-4a6f-b7c7-ddb0529d0844-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:39.086056 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.085971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a800ac6-982c-486f-9c47-e57a2a34770b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:39.086056 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.085992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckt6g\" (UniqueName: \"kubernetes.io/projected/5a800ac6-982c-486f-9c47-e57a2a34770b-kube-api-access-ckt6g\") pod \"payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:39.086955 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.086930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a800ac6-982c-486f-9c47-e57a2a34770b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:39.087002 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.086956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eed963-6f72-4a6f-b7c7-ddb0529d0844-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:39.099793 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.099756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6w54\" (UniqueName: \"kubernetes.io/projected/88eed963-6f72-4a6f-b7c7-ddb0529d0844-kube-api-access-j6w54\") pod \"payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:39.099793 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.099755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckt6g\" (UniqueName: \"kubernetes.io/projected/5a800ac6-982c-486f-9c47-e57a2a34770b-kube-api-access-ckt6g\") pod \"payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:39.220231 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.220143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:39.274699 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.274638    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:39.491991 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.491962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"]
Sep 11 16:09:39.493710 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:09:39.493678    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eed963_6f72_4a6f_b7c7_ddb0529d0844.slice/crio-2a8615581542872183dba31d8dae1b98335b66a4238607f39c8284781eef7d15 WatchSource:0}: Error finding container 2a8615581542872183dba31d8dae1b98335b66a4238607f39c8284781eef7d15: Status 404 returned error can't find the container with id 2a8615581542872183dba31d8dae1b98335b66a4238607f39c8284781eef7d15
Sep 11 16:09:39.572673 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:39.566534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"]
Sep 11 16:09:40.174804 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.174763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx" event={"ID":"5a800ac6-982c-486f-9c47-e57a2a34770b","Type":"ContainerStarted","Data":"c670244abcc1bfe22ec4e3b3a789534465ae15bceb2405a939e9bdbea1bc405e"}
Sep 11 16:09:40.175287 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.174831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx" event={"ID":"5a800ac6-982c-486f-9c47-e57a2a34770b","Type":"ContainerStarted","Data":"b4c8cf48fcb41047cf482357a05a035c8f2edb8513425c28772f270894a7445d"}
Sep 11 16:09:40.175287 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.174891    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:40.176795 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.176766    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f" event={"ID":"88eed963-6f72-4a6f-b7c7-ddb0529d0844","Type":"ContainerStarted","Data":"11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a"}
Sep 11 16:09:40.176927 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.176813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f" event={"ID":"88eed963-6f72-4a6f-b7c7-ddb0529d0844","Type":"ContainerStarted","Data":"2a8615581542872183dba31d8dae1b98335b66a4238607f39c8284781eef7d15"}
Sep 11 16:09:40.176927 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.176859    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:40.196779 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.196719    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx" podStartSLOduration=2.1967028 podStartE2EDuration="2.1967028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:40.194363775 +0000 UTC m=+876.618181493" watchObservedRunningTime="2026-09-11 16:09:40.1967028 +0000 UTC m=+876.620520485"
Sep 11 16:09:40.213726 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:40.213660    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f" podStartSLOduration=2.213636745 podStartE2EDuration="2.213636745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:40.211976477 +0000 UTC m=+876.635794199" watchObservedRunningTime="2026-09-11 16:09:40.213636745 +0000 UTC m=+876.637454443"
Sep 11 16:09:48.638673 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.638633    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"]
Sep 11 16:09:48.642005 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.641986    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:48.645285 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.645246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a6560b69-dockercfg-pzz9p\""
Sep 11 16:09:48.645456 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.645438    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a6560b69\""
Sep 11 16:09:48.653429 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.653402    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"]
Sep 11 16:09:48.702738 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.702700    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"]
Sep 11 16:09:48.706307 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.706286    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:48.717686 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.717655    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"]
Sep 11 16:09:48.782316 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.782276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048ffc4e-16ef-4c02-ac21-74f79585c916-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6560b69-5756795985-9qznj\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") " pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:48.782316 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.782313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzk7w\" (UniqueName: \"kubernetes.io/projected/048ffc4e-16ef-4c02-ac21-74f79585c916-kube-api-access-mzk7w\") pod \"payload-processing-e2e-ait-a6560b69-5756795985-9qznj\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") " pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:48.782530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.782374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2n9q\" (UniqueName: \"kubernetes.io/projected/2c28f689-a1a2-437d-8748-1041f9b0ad9a-kube-api-access-q2n9q\") pod \"payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:48.782530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.782503    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c28f689-a1a2-437d-8748-1041f9b0ad9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:48.883766 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.883720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2n9q\" (UniqueName: \"kubernetes.io/projected/2c28f689-a1a2-437d-8748-1041f9b0ad9a-kube-api-access-q2n9q\") pod \"payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:48.883938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.883805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c28f689-a1a2-437d-8748-1041f9b0ad9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:48.883938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.883829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048ffc4e-16ef-4c02-ac21-74f79585c916-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6560b69-5756795985-9qznj\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") " pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:48.883938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.883845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzk7w\" (UniqueName: \"kubernetes.io/projected/048ffc4e-16ef-4c02-ac21-74f79585c916-kube-api-access-mzk7w\") pod \"payload-processing-e2e-ait-a6560b69-5756795985-9qznj\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") " pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:48.884832 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.884807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c28f689-a1a2-437d-8748-1041f9b0ad9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:48.884880 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.884807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048ffc4e-16ef-4c02-ac21-74f79585c916-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6560b69-5756795985-9qznj\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") " pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:48.895420 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.895344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzk7w\" (UniqueName: \"kubernetes.io/projected/048ffc4e-16ef-4c02-ac21-74f79585c916-kube-api-access-mzk7w\") pod \"payload-processing-e2e-ait-a6560b69-5756795985-9qznj\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") " pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:48.895420 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.895344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2n9q\" (UniqueName: \"kubernetes.io/projected/2c28f689-a1a2-437d-8748-1041f9b0ad9a-kube-api-access-q2n9q\") pod \"payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:48.953553 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:48.953514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:49.017587 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.017557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:49.228052 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.228027    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"]
Sep 11 16:09:49.230339 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:09:49.230297    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048ffc4e_16ef_4c02_ac21_74f79585c916.slice/crio-1b896a33ca14f0043393a9ccdbff72ddd946226a8cd7a7adad68b41c3ecd80a7 WatchSource:0}: Error finding container 1b896a33ca14f0043393a9ccdbff72ddd946226a8cd7a7adad68b41c3ecd80a7: Status 404 returned error can't find the container with id 1b896a33ca14f0043393a9ccdbff72ddd946226a8cd7a7adad68b41c3ecd80a7
Sep 11 16:09:49.303579 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.303544    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"]
Sep 11 16:09:49.307459 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:09:49.307431    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c28f689_a1a2_437d_8748_1041f9b0ad9a.slice/crio-43d9f29ab6d25cbaf13877dbb073ebf02f59f6306f085563d1b42936933b4395 WatchSource:0}: Error finding container 43d9f29ab6d25cbaf13877dbb073ebf02f59f6306f085563d1b42936933b4395: Status 404 returned error can't find the container with id 43d9f29ab6d25cbaf13877dbb073ebf02f59f6306f085563d1b42936933b4395
Sep 11 16:09:49.795807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.795767    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"]
Sep 11 16:09:49.804050 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.801345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:49.805647 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.805619    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9c5b09-dockercfg-vfk8h\""
Sep 11 16:09:49.806386 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.806368    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9c5b09\""
Sep 11 16:09:49.811956 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.811930    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"]
Sep 11 16:09:49.894283 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.894229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjhhh\" (UniqueName: \"kubernetes.io/projected/45921c6b-aa86-491a-844b-acf0f0e9ee0d-kube-api-access-rjhhh\") pod \"payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:49.894461 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.894412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45921c6b-aa86-491a-844b-acf0f0e9ee0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:49.902918 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.902884    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"]
Sep 11 16:09:49.906532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.906515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:49.920228 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.920189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"]
Sep 11 16:09:49.995626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.995580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45921c6b-aa86-491a-844b-acf0f0e9ee0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:49.995837 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.995638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvfd\" (UniqueName: \"kubernetes.io/projected/3cb20e12-852b-4b71-87c9-25aef938bbe1-kube-api-access-fvvfd\") pod \"payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") " pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:49.995837 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.995677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb20e12-852b-4b71-87c9-25aef938bbe1-plugins-config-volume\") pod \"payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") " pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:49.995837 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.995795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjhhh\" (UniqueName: \"kubernetes.io/projected/45921c6b-aa86-491a-844b-acf0f0e9ee0d-kube-api-access-rjhhh\") pod \"payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:49.996576 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:49.996547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45921c6b-aa86-491a-844b-acf0f0e9ee0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:50.004856 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.004821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjhhh\" (UniqueName: \"kubernetes.io/projected/45921c6b-aa86-491a-844b-acf0f0e9ee0d-kube-api-access-rjhhh\") pod \"payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:50.097165 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.097069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvfd\" (UniqueName: \"kubernetes.io/projected/3cb20e12-852b-4b71-87c9-25aef938bbe1-kube-api-access-fvvfd\") pod \"payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") " pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:50.097165 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.097139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb20e12-852b-4b71-87c9-25aef938bbe1-plugins-config-volume\") pod \"payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") " pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:50.097874 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.097852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb20e12-852b-4b71-87c9-25aef938bbe1-plugins-config-volume\") pod \"payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") " pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:50.106609 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.106573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvfd\" (UniqueName: \"kubernetes.io/projected/3cb20e12-852b-4b71-87c9-25aef938bbe1-kube-api-access-fvvfd\") pod \"payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") " pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:50.115532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.115502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:50.221066 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.221036    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:50.227498 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.227449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr" event={"ID":"2c28f689-a1a2-437d-8748-1041f9b0ad9a","Type":"ContainerStarted","Data":"e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3"}
Sep 11 16:09:50.227665 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.227525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:09:50.227665 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.227547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr" event={"ID":"2c28f689-a1a2-437d-8748-1041f9b0ad9a","Type":"ContainerStarted","Data":"43d9f29ab6d25cbaf13877dbb073ebf02f59f6306f085563d1b42936933b4395"}
Sep 11 16:09:50.229316 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.229277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj" event={"ID":"048ffc4e-16ef-4c02-ac21-74f79585c916","Type":"ContainerStarted","Data":"88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f"}
Sep 11 16:09:50.229433 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.229350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj" event={"ID":"048ffc4e-16ef-4c02-ac21-74f79585c916","Type":"ContainerStarted","Data":"1b896a33ca14f0043393a9ccdbff72ddd946226a8cd7a7adad68b41c3ecd80a7"}
Sep 11 16:09:50.229433 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.229381    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:09:50.248750 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.248698    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr" podStartSLOduration=2.24867942 podStartE2EDuration="2.24867942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:50.246444935 +0000 UTC m=+886.670262646" watchObservedRunningTime="2026-09-11 16:09:50.24867942 +0000 UTC m=+886.672497115"
Sep 11 16:09:50.271161 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.271098    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj" podStartSLOduration=2.27105648 podStartE2EDuration="2.27105648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:50.267930288 +0000 UTC m=+886.691747994" watchObservedRunningTime="2026-09-11 16:09:50.27105648 +0000 UTC m=+886.694874174"
Sep 11 16:09:50.389512 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.389485    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"]
Sep 11 16:09:50.391865 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:09:50.391836    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45921c6b_aa86_491a_844b_acf0f0e9ee0d.slice/crio-0ef5e1b2180a97884598089679b994e1d6f74823367cafd12b2fe08343446626 WatchSource:0}: Error finding container 0ef5e1b2180a97884598089679b994e1d6f74823367cafd12b2fe08343446626: Status 404 returned error can't find the container with id 0ef5e1b2180a97884598089679b994e1d6f74823367cafd12b2fe08343446626
Sep 11 16:09:50.502073 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:50.502046    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"]
Sep 11 16:09:50.504049 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:09:50.504013    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb20e12_852b_4b71_87c9_25aef938bbe1.slice/crio-422d5801e0a3977e4cb42083d4a2a5d1dd2f3ed3de2225834377986903b250a3 WatchSource:0}: Error finding container 422d5801e0a3977e4cb42083d4a2a5d1dd2f3ed3de2225834377986903b250a3: Status 404 returned error can't find the container with id 422d5801e0a3977e4cb42083d4a2a5d1dd2f3ed3de2225834377986903b250a3
Sep 11 16:09:51.183452 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.183418    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:09:51.183915 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.183505    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:09:51.239102 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.239063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2" event={"ID":"45921c6b-aa86-491a-844b-acf0f0e9ee0d","Type":"ContainerStarted","Data":"7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5"}
Sep 11 16:09:51.239319 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.239135    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:09:51.239319 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.239161    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2" event={"ID":"45921c6b-aa86-491a-844b-acf0f0e9ee0d","Type":"ContainerStarted","Data":"0ef5e1b2180a97884598089679b994e1d6f74823367cafd12b2fe08343446626"}
Sep 11 16:09:51.241307 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.241277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm" event={"ID":"3cb20e12-852b-4b71-87c9-25aef938bbe1","Type":"ContainerStarted","Data":"c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576"}
Sep 11 16:09:51.241435 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.241331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm" event={"ID":"3cb20e12-852b-4b71-87c9-25aef938bbe1","Type":"ContainerStarted","Data":"422d5801e0a3977e4cb42083d4a2a5d1dd2f3ed3de2225834377986903b250a3"}
Sep 11 16:09:51.241515 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.241436    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:09:51.261266 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.261195    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2" podStartSLOduration=2.26117639 podStartE2EDuration="2.26117639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:51.258326408 +0000 UTC m=+887.682144123" watchObservedRunningTime="2026-09-11 16:09:51.26117639 +0000 UTC m=+887.684994084"
Sep 11 16:09:51.281362 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:09:51.281297    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm" podStartSLOduration=2.281273425 podStartE2EDuration="2.281273425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:51.279847763 +0000 UTC m=+887.703665485" watchObservedRunningTime="2026-09-11 16:09:51.281273425 +0000 UTC m=+887.705091119"
Sep 11 16:10:00.487399 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.487312    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"]
Sep 11 16:10:00.490948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.490923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.494653 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.494628    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2b1bd5\""
Sep 11 16:10:00.510698 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.510660    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"]
Sep 11 16:10:00.514468 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.514435    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"]
Sep 11 16:10:00.518742 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.518717    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:00.521630 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.521606    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2b1bd5-dockercfg-lk4j4\""
Sep 11 16:10:00.532376 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.532337    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"]
Sep 11 16:10:00.599634 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.599592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd68w\" (UniqueName: \"kubernetes.io/projected/ff51ce42-e695-433d-a2d1-eca44abe0f00-kube-api-access-dd68w\") pod \"payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.599809 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.599692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be3b9-5d82-4058-a012-89c07626db0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:00.599809 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.599729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff51ce42-e695-433d-a2d1-eca44abe0f00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.599809 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.599760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2s29\" (UniqueName: \"kubernetes.io/projected/8e8be3b9-5d82-4058-a012-89c07626db0e-kube-api-access-w2s29\") pod \"payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:00.700610 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.700549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be3b9-5d82-4058-a012-89c07626db0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:00.700610 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.700608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff51ce42-e695-433d-a2d1-eca44abe0f00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.700843 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.700636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2s29\" (UniqueName: \"kubernetes.io/projected/8e8be3b9-5d82-4058-a012-89c07626db0e-kube-api-access-w2s29\") pod \"payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:00.700843 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.700690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd68w\" (UniqueName: \"kubernetes.io/projected/ff51ce42-e695-433d-a2d1-eca44abe0f00-kube-api-access-dd68w\") pod \"payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.701388 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.701366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be3b9-5d82-4058-a012-89c07626db0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:00.701467 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.701374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff51ce42-e695-433d-a2d1-eca44abe0f00-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.709541 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.709515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2s29\" (UniqueName: \"kubernetes.io/projected/8e8be3b9-5d82-4058-a012-89c07626db0e-kube-api-access-w2s29\") pod \"payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:00.709677 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.709544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd68w\" (UniqueName: \"kubernetes.io/projected/ff51ce42-e695-433d-a2d1-eca44abe0f00-kube-api-access-dd68w\") pod \"payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.801709 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.801655    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:00.830107 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:00.830074    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:01.091645 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.091613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"]
Sep 11 16:10:01.092309 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:10:01.092280    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff51ce42_e695_433d_a2d1_eca44abe0f00.slice/crio-ccbc5cb665c4f56f2d743b43430146f3a3572b5d6f7db7984f3528e4d1d508c4 WatchSource:0}: Error finding container ccbc5cb665c4f56f2d743b43430146f3a3572b5d6f7db7984f3528e4d1d508c4: Status 404 returned error can't find the container with id ccbc5cb665c4f56f2d743b43430146f3a3572b5d6f7db7984f3528e4d1d508c4
Sep 11 16:10:01.134649 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.134626    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"]
Sep 11 16:10:01.151480 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:10:01.151452    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8be3b9_5d82_4058_a012_89c07626db0e.slice/crio-ff8d28faf35f892b5f56e1da9a9454517e84e998fa03cc924e9c2919ee1dcd4f WatchSource:0}: Error finding container ff8d28faf35f892b5f56e1da9a9454517e84e998fa03cc924e9c2919ee1dcd4f: Status 404 returned error can't find the container with id ff8d28faf35f892b5f56e1da9a9454517e84e998fa03cc924e9c2919ee1dcd4f
Sep 11 16:10:01.243416 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.243383    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:10:01.243770 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.243741    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:10:01.294814 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.294297    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2" event={"ID":"ff51ce42-e695-433d-a2d1-eca44abe0f00","Type":"ContainerStarted","Data":"876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718"}
Sep 11 16:10:01.294814 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.294362    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2" event={"ID":"ff51ce42-e695-433d-a2d1-eca44abe0f00","Type":"ContainerStarted","Data":"ccbc5cb665c4f56f2d743b43430146f3a3572b5d6f7db7984f3528e4d1d508c4"}
Sep 11 16:10:01.294814 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.294554    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:01.296329 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.296291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t" event={"ID":"8e8be3b9-5d82-4058-a012-89c07626db0e","Type":"ContainerStarted","Data":"ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce"}
Sep 11 16:10:01.296453 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.296341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t" event={"ID":"8e8be3b9-5d82-4058-a012-89c07626db0e","Type":"ContainerStarted","Data":"ff8d28faf35f892b5f56e1da9a9454517e84e998fa03cc924e9c2919ee1dcd4f"}
Sep 11 16:10:01.296453 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.296399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:01.316767 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.316416    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2" podStartSLOduration=1.316393393 podStartE2EDuration="1.316393393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:01.315916789 +0000 UTC m=+897.739734474" watchObservedRunningTime="2026-09-11 16:10:01.316393393 +0000 UTC m=+897.740211087"
Sep 11 16:10:01.340327 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:01.340198    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t" podStartSLOduration=1.340178025 podStartE2EDuration="1.340178025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:01.337579366 +0000 UTC m=+897.761397084" watchObservedRunningTime="2026-09-11 16:10:01.340178025 +0000 UTC m=+897.763995720"
Sep 11 16:10:02.247933 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:02.247899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:10:02.248455 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:02.247986    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:10:11.302510 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.302467    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"]
Sep 11 16:10:11.302984 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.302718    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj" podUID="048ffc4e-16ef-4c02-ac21-74f79585c916" containerName="payload-processing" containerID="cri-o://88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f" gracePeriod=30
Sep 11 16:10:11.312778 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.312727    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"]
Sep 11 16:10:11.313326 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.313270    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr" podUID="2c28f689-a1a2-437d-8748-1041f9b0ad9a" containerName="payload-pre-processing" containerID="cri-o://e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3" gracePeriod=30
Sep 11 16:10:11.582110 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.582086    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:10:11.585322 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.585300    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:10:11.706856 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.706819    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzk7w\" (UniqueName: \"kubernetes.io/projected/048ffc4e-16ef-4c02-ac21-74f79585c916-kube-api-access-mzk7w\") pod \"048ffc4e-16ef-4c02-ac21-74f79585c916\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") "
Sep 11 16:10:11.706856 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.706854    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c28f689-a1a2-437d-8748-1041f9b0ad9a-plugins-config-volume\") pod \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") "
Sep 11 16:10:11.707101 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.706900    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2n9q\" (UniqueName: \"kubernetes.io/projected/2c28f689-a1a2-437d-8748-1041f9b0ad9a-kube-api-access-q2n9q\") pod \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\" (UID: \"2c28f689-a1a2-437d-8748-1041f9b0ad9a\") "
Sep 11 16:10:11.707101 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.706930    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048ffc4e-16ef-4c02-ac21-74f79585c916-plugins-config-volume\") pod \"048ffc4e-16ef-4c02-ac21-74f79585c916\" (UID: \"048ffc4e-16ef-4c02-ac21-74f79585c916\") "
Sep 11 16:10:11.707416 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.707384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c28f689-a1a2-437d-8748-1041f9b0ad9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c28f689-a1a2-437d-8748-1041f9b0ad9a" (UID: "2c28f689-a1a2-437d-8748-1041f9b0ad9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:10:11.707484 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.707386    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048ffc4e-16ef-4c02-ac21-74f79585c916-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "048ffc4e-16ef-4c02-ac21-74f79585c916" (UID: "048ffc4e-16ef-4c02-ac21-74f79585c916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:10:11.709182 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.709157    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28f689-a1a2-437d-8748-1041f9b0ad9a-kube-api-access-q2n9q" (OuterVolumeSpecName: "kube-api-access-q2n9q") pod "2c28f689-a1a2-437d-8748-1041f9b0ad9a" (UID: "2c28f689-a1a2-437d-8748-1041f9b0ad9a"). InnerVolumeSpecName "kube-api-access-q2n9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:11.709259 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.709194    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048ffc4e-16ef-4c02-ac21-74f79585c916-kube-api-access-mzk7w" (OuterVolumeSpecName: "kube-api-access-mzk7w") pod "048ffc4e-16ef-4c02-ac21-74f79585c916" (UID: "048ffc4e-16ef-4c02-ac21-74f79585c916"). InnerVolumeSpecName "kube-api-access-mzk7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:11.808492 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.808401    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzk7w\" (UniqueName: \"kubernetes.io/projected/048ffc4e-16ef-4c02-ac21-74f79585c916-kube-api-access-mzk7w\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.808492 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.808436    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c28f689-a1a2-437d-8748-1041f9b0ad9a-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.808492 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.808446    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2n9q\" (UniqueName: \"kubernetes.io/projected/2c28f689-a1a2-437d-8748-1041f9b0ad9a-kube-api-access-q2n9q\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.808492 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:11.808454    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048ffc4e-16ef-4c02-ac21-74f79585c916-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:10:12.306309 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.306277    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:10:12.306702 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.306357    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:10:12.373539 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.373498    2559 generic.go:356] "Generic (PLEG): container finished" podID="048ffc4e-16ef-4c02-ac21-74f79585c916" containerID="88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f" exitCode=0
Sep 11 16:10:12.373718 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.373558    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"
Sep 11 16:10:12.373718 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.373546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj" event={"ID":"048ffc4e-16ef-4c02-ac21-74f79585c916","Type":"ContainerDied","Data":"88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f"}
Sep 11 16:10:12.373824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.373728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj" event={"ID":"048ffc4e-16ef-4c02-ac21-74f79585c916","Type":"ContainerDied","Data":"1b896a33ca14f0043393a9ccdbff72ddd946226a8cd7a7adad68b41c3ecd80a7"}
Sep 11 16:10:12.373824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.373762    2559 scope.go:117] "RemoveContainer" containerID="88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f"
Sep 11 16:10:12.375471 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.375360    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c28f689-a1a2-437d-8748-1041f9b0ad9a" containerID="e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3" exitCode=0
Sep 11 16:10:12.375471 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.375424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr" event={"ID":"2c28f689-a1a2-437d-8748-1041f9b0ad9a","Type":"ContainerDied","Data":"e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3"}
Sep 11 16:10:12.375471 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.375468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr" event={"ID":"2c28f689-a1a2-437d-8748-1041f9b0ad9a","Type":"ContainerDied","Data":"43d9f29ab6d25cbaf13877dbb073ebf02f59f6306f085563d1b42936933b4395"}
Sep 11 16:10:12.375976 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.375951    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"
Sep 11 16:10:12.383958 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.383933    2559 scope.go:117] "RemoveContainer" containerID="88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f"
Sep 11 16:10:12.384345 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:10:12.384313    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f\": container with ID starting with 88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f not found: ID does not exist" containerID="88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f"
Sep 11 16:10:12.384440 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.384359    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f"} err="failed to get container status \"88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f\": rpc error: code = NotFound desc = could not find container \"88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f\": container with ID starting with 88013bb6051d653f48ccd4b259f832f6546adc06582aa4affbd2dcf33f39534f not found: ID does not exist"
Sep 11 16:10:12.384440 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.384386    2559 scope.go:117] "RemoveContainer" containerID="e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3"
Sep 11 16:10:12.393517 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.393310    2559 scope.go:117] "RemoveContainer" containerID="e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3"
Sep 11 16:10:12.394016 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:10:12.393987    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3\": container with ID starting with e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3 not found: ID does not exist" containerID="e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3"
Sep 11 16:10:12.394129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.394023    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3"} err="failed to get container status \"e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3\": rpc error: code = NotFound desc = could not find container \"e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3\": container with ID starting with e4059801c107eff68c48467af02022650a954ac3364ee8442b8fee023983c3c3 not found: ID does not exist"
Sep 11 16:10:12.396874 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.396848    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"]
Sep 11 16:10:12.404691 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.404661    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6560b69-5756795985-9qznj"]
Sep 11 16:10:12.420070 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.420035    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"]
Sep 11 16:10:12.426698 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:12.426667    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6560b69-8678d75774-dp6pr"]
Sep 11 16:10:14.035281 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:14.035222    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="048ffc4e-16ef-4c02-ac21-74f79585c916" path="/var/lib/kubelet/pods/048ffc4e-16ef-4c02-ac21-74f79585c916/volumes"
Sep 11 16:10:14.035643 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:14.035624    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c28f689-a1a2-437d-8748-1041f9b0ad9a" path="/var/lib/kubelet/pods/2c28f689-a1a2-437d-8748-1041f9b0ad9a/volumes"
Sep 11 16:10:18.761441 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.761404    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"]
Sep 11 16:10:18.761834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.761796    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048ffc4e-16ef-4c02-ac21-74f79585c916" containerName="payload-processing"
Sep 11 16:10:18.761834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.761807    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="048ffc4e-16ef-4c02-ac21-74f79585c916" containerName="payload-processing"
Sep 11 16:10:18.761834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.761829    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c28f689-a1a2-437d-8748-1041f9b0ad9a" containerName="payload-pre-processing"
Sep 11 16:10:18.761834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.761834    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c28f689-a1a2-437d-8748-1041f9b0ad9a" containerName="payload-pre-processing"
Sep 11 16:10:18.761953 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.761911    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c28f689-a1a2-437d-8748-1041f9b0ad9a" containerName="payload-pre-processing"
Sep 11 16:10:18.761953 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.761923    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="048ffc4e-16ef-4c02-ac21-74f79585c916" containerName="payload-processing"
Sep 11 16:10:18.766606 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.766588    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:18.770860 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.770839    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f8b85f0c-dockercfg-2gqkk\""
Sep 11 16:10:18.771459 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.771441    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f8b85f0c\""
Sep 11 16:10:18.785755 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.785726    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"]
Sep 11 16:10:18.865764 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.865725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87cx6\" (UniqueName: \"kubernetes.io/projected/78230b76-f805-4342-86fc-aacd2152e063-kube-api-access-87cx6\") pod \"payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:18.865920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.865777    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78230b76-f805-4342-86fc-aacd2152e063-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:18.966824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.966788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87cx6\" (UniqueName: \"kubernetes.io/projected/78230b76-f805-4342-86fc-aacd2152e063-kube-api-access-87cx6\") pod \"payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:18.966946 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.966836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78230b76-f805-4342-86fc-aacd2152e063-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:18.967525 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.967507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78230b76-f805-4342-86fc-aacd2152e063-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:18.982887 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:18.982853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87cx6\" (UniqueName: \"kubernetes.io/projected/78230b76-f805-4342-86fc-aacd2152e063-kube-api-access-87cx6\") pod \"payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:19.076322 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:19.076208    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:19.380409 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:19.380383    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"]
Sep 11 16:10:19.382395 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:10:19.382366    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78230b76_f805_4342_86fc_aacd2152e063.slice/crio-fde63ddcde78fcedd3ca319c79d56ab719bb8396e7453d36cb790b26163e755f WatchSource:0}: Error finding container fde63ddcde78fcedd3ca319c79d56ab719bb8396e7453d36cb790b26163e755f: Status 404 returned error can't find the container with id fde63ddcde78fcedd3ca319c79d56ab719bb8396e7453d36cb790b26163e755f
Sep 11 16:10:19.414917 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:19.414879    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk" event={"ID":"78230b76-f805-4342-86fc-aacd2152e063","Type":"ContainerStarted","Data":"fde63ddcde78fcedd3ca319c79d56ab719bb8396e7453d36cb790b26163e755f"}
Sep 11 16:10:20.421771 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:20.421736    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk" event={"ID":"78230b76-f805-4342-86fc-aacd2152e063","Type":"ContainerStarted","Data":"1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21"}
Sep 11 16:10:20.422222 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:20.421802    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:20.468969 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:20.468914    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk" podStartSLOduration=2.468895163 podStartE2EDuration="2.468895163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:20.466068452 +0000 UTC m=+916.889886151" watchObservedRunningTime="2026-09-11 16:10:20.468895163 +0000 UTC m=+916.892712857"
Sep 11 16:10:31.429826 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:31.429792    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:47.267888 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.267852    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"]
Sep 11 16:10:47.270713 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.268114    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk" podUID="78230b76-f805-4342-86fc-aacd2152e063" containerName="payload-processing" containerID="cri-o://1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21" gracePeriod=30
Sep 11 16:10:47.523919 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.523857    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:47.557693 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.557656    2559 generic.go:356] "Generic (PLEG): container finished" podID="78230b76-f805-4342-86fc-aacd2152e063" containerID="1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21" exitCode=0
Sep 11 16:10:47.557857 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.557699    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"
Sep 11 16:10:47.557857 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.557745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk" event={"ID":"78230b76-f805-4342-86fc-aacd2152e063","Type":"ContainerDied","Data":"1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21"}
Sep 11 16:10:47.557857 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.557796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk" event={"ID":"78230b76-f805-4342-86fc-aacd2152e063","Type":"ContainerDied","Data":"fde63ddcde78fcedd3ca319c79d56ab719bb8396e7453d36cb790b26163e755f"}
Sep 11 16:10:47.557857 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.557820    2559 scope.go:117] "RemoveContainer" containerID="1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21"
Sep 11 16:10:47.566487 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.566470    2559 scope.go:117] "RemoveContainer" containerID="1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21"
Sep 11 16:10:47.566779 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:10:47.566760    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21\": container with ID starting with 1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21 not found: ID does not exist" containerID="1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21"
Sep 11 16:10:47.566852 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.566788    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21"} err="failed to get container status \"1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21\": rpc error: code = NotFound desc = could not find container \"1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21\": container with ID starting with 1f46ed7243bca30dfbf99d015e8e78051699279f2d84fb291427a1ee0f7d6d21 not found: ID does not exist"
Sep 11 16:10:47.632051 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.632020    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87cx6\" (UniqueName: \"kubernetes.io/projected/78230b76-f805-4342-86fc-aacd2152e063-kube-api-access-87cx6\") pod \"78230b76-f805-4342-86fc-aacd2152e063\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") "
Sep 11 16:10:47.632051 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.632055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78230b76-f805-4342-86fc-aacd2152e063-plugins-config-volume\") pod \"78230b76-f805-4342-86fc-aacd2152e063\" (UID: \"78230b76-f805-4342-86fc-aacd2152e063\") "
Sep 11 16:10:47.632605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.632571    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78230b76-f805-4342-86fc-aacd2152e063-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78230b76-f805-4342-86fc-aacd2152e063" (UID: "78230b76-f805-4342-86fc-aacd2152e063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:10:47.634184 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.634162    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78230b76-f805-4342-86fc-aacd2152e063-kube-api-access-87cx6" (OuterVolumeSpecName: "kube-api-access-87cx6") pod "78230b76-f805-4342-86fc-aacd2152e063" (UID: "78230b76-f805-4342-86fc-aacd2152e063"). InnerVolumeSpecName "kube-api-access-87cx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:47.733519 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.733478    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87cx6\" (UniqueName: \"kubernetes.io/projected/78230b76-f805-4342-86fc-aacd2152e063-kube-api-access-87cx6\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.733519 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.733512    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78230b76-f805-4342-86fc-aacd2152e063-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.894278 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.894226    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"]
Sep 11 16:10:47.901534 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:47.901503    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8b85f0c-698865f958-cj7qk"]
Sep 11 16:10:48.034129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:48.034095    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78230b76-f805-4342-86fc-aacd2152e063" path="/var/lib/kubelet/pods/78230b76-f805-4342-86fc-aacd2152e063/volumes"
Sep 11 16:10:55.856379 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:55.856339    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"]
Sep 11 16:10:55.856853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:55.856745    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78230b76-f805-4342-86fc-aacd2152e063" containerName="payload-processing"
Sep 11 16:10:55.856853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:55.856757    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="78230b76-f805-4342-86fc-aacd2152e063" containerName="payload-processing"
Sep 11 16:10:55.856853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:55.856836    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="78230b76-f805-4342-86fc-aacd2152e063" containerName="payload-processing"
Sep 11 16:10:55.861734 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:55.861715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:55.864697 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:55.864675    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-20aaef10\""
Sep 11 16:10:55.873389 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:55.873362    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"]
Sep 11 16:10:56.016076 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.016033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f289\" (UniqueName: \"kubernetes.io/projected/faa435d5-804c-426d-82ea-f238392ff698-kube-api-access-8f289\") pod \"payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") " pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.016307 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.016105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa435d5-804c-426d-82ea-f238392ff698-plugins-config-volume\") pod \"payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") " pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.117364 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.117264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f289\" (UniqueName: \"kubernetes.io/projected/faa435d5-804c-426d-82ea-f238392ff698-kube-api-access-8f289\") pod \"payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") " pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.117364 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.117361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa435d5-804c-426d-82ea-f238392ff698-plugins-config-volume\") pod \"payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") " pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.118058 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.118037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa435d5-804c-426d-82ea-f238392ff698-plugins-config-volume\") pod \"payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") " pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.128003 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.127973    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f289\" (UniqueName: \"kubernetes.io/projected/faa435d5-804c-426d-82ea-f238392ff698-kube-api-access-8f289\") pod \"payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") " pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.172596 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.172551    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.449520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.449492    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"]
Sep 11 16:10:56.451809 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:10:56.451777    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa435d5_804c_426d_82ea_f238392ff698.slice/crio-63aed731c4c2dd75cb901d4cdb5cf5c420d679d2f686a4a740eac9cea99f51c2 WatchSource:0}: Error finding container 63aed731c4c2dd75cb901d4cdb5cf5c420d679d2f686a4a740eac9cea99f51c2: Status 404 returned error can't find the container with id 63aed731c4c2dd75cb901d4cdb5cf5c420d679d2f686a4a740eac9cea99f51c2
Sep 11 16:10:56.612758 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.612720    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk" event={"ID":"faa435d5-804c-426d-82ea-f238392ff698","Type":"ContainerStarted","Data":"89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd"}
Sep 11 16:10:56.612942 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.612805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk" event={"ID":"faa435d5-804c-426d-82ea-f238392ff698","Type":"ContainerStarted","Data":"63aed731c4c2dd75cb901d4cdb5cf5c420d679d2f686a4a740eac9cea99f51c2"}
Sep 11 16:10:56.612942 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.612854    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:10:56.642866 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:10:56.642806    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk" podStartSLOduration=1.64279022 podStartE2EDuration="1.64279022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:56.64085485 +0000 UTC m=+953.064672564" watchObservedRunningTime="2026-09-11 16:10:56.64279022 +0000 UTC m=+953.066607950"
Sep 11 16:11:07.621449 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:07.621407    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:11:25.498351 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.498313    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"]
Sep 11 16:11:25.503250 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.503219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.506054 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.506033    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-497b3375-kserve-self-signed-certs\""
Sep 11 16:11:25.512645 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.512619    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"]
Sep 11 16:11:25.588369 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.588221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tmp-dir\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.588369 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.588355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphxz\" (UniqueName: \"kubernetes.io/projected/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-kube-api-access-wphxz\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.588605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.588420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tls-certs\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.588605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.588477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-dshm\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.588605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.588499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-model-cache\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.588605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.588557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-home\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.689688 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.689649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphxz\" (UniqueName: \"kubernetes.io/projected/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-kube-api-access-wphxz\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.689867 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.689716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tls-certs\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.689867 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.689752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-dshm\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.689867 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.689781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-model-cache\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.689867 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.689832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-home\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.690060 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.689871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tmp-dir\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.690294 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.690273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-model-cache\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.690393 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.690305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tmp-dir\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.690393 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.690352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-home\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.692344 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.692313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-dshm\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.692454 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.692428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tls-certs\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.699455 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.699431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphxz\" (UniqueName: \"kubernetes.io/projected/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-kube-api-access-wphxz\") pod \"e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") " pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:25.812732 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:25.812636    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:26.096589 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:26.096554    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"]
Sep 11 16:11:26.098218 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:11:26.098193    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5a3577f_8dd1_4b3d_97a0_8aa0fa7ae7ab.slice/crio-b45ecdd81d89d250fb980a81f61d8cc2e30810f46962c927a12218da99bfbdbb WatchSource:0}: Error finding container b45ecdd81d89d250fb980a81f61d8cc2e30810f46962c927a12218da99bfbdbb: Status 404 returned error can't find the container with id b45ecdd81d89d250fb980a81f61d8cc2e30810f46962c927a12218da99bfbdbb
Sep 11 16:11:26.767432 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:26.767393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt" event={"ID":"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab","Type":"ContainerStarted","Data":"3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835"}
Sep 11 16:11:26.767892 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:26.767456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt" event={"ID":"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab","Type":"ContainerStarted","Data":"b45ecdd81d89d250fb980a81f61d8cc2e30810f46962c927a12218da99bfbdbb"}
Sep 11 16:11:26.786432 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:26.786379    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt" podStartSLOduration=1.7863636550000002 podStartE2EDuration="1.786363655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:26.785909847 +0000 UTC m=+983.209727567" watchObservedRunningTime="2026-09-11 16:11:26.786363655 +0000 UTC m=+983.210181406"
Sep 11 16:11:33.539562 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.539528    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"]
Sep 11 16:11:33.540032 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.539786    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk" podUID="faa435d5-804c-426d-82ea-f238392ff698" containerName="payload-processing" containerID="cri-o://89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd" gracePeriod=30
Sep 11 16:11:33.794818 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.794750    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:11:33.807900 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.807869    2559 generic.go:356] "Generic (PLEG): container finished" podID="faa435d5-804c-426d-82ea-f238392ff698" containerID="89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd" exitCode=0
Sep 11 16:11:33.808065 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.807917    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"
Sep 11 16:11:33.808065 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.807941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk" event={"ID":"faa435d5-804c-426d-82ea-f238392ff698","Type":"ContainerDied","Data":"89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd"}
Sep 11 16:11:33.808065 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.807981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk" event={"ID":"faa435d5-804c-426d-82ea-f238392ff698","Type":"ContainerDied","Data":"63aed731c4c2dd75cb901d4cdb5cf5c420d679d2f686a4a740eac9cea99f51c2"}
Sep 11 16:11:33.808065 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.808002    2559 scope.go:117] "RemoveContainer" containerID="89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd"
Sep 11 16:11:33.817426 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.817398    2559 scope.go:117] "RemoveContainer" containerID="89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd"
Sep 11 16:11:33.817752 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:11:33.817727    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd\": container with ID starting with 89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd not found: ID does not exist" containerID="89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd"
Sep 11 16:11:33.817855 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.817762    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd"} err="failed to get container status \"89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd\": rpc error: code = NotFound desc = could not find container \"89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd\": container with ID starting with 89e1b352113b4192ac24ab48d1ae80cff7202c8bd10198e27ba4009b004c82cd not found: ID does not exist"
Sep 11 16:11:33.870132 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.870086    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f289\" (UniqueName: \"kubernetes.io/projected/faa435d5-804c-426d-82ea-f238392ff698-kube-api-access-8f289\") pod \"faa435d5-804c-426d-82ea-f238392ff698\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") "
Sep 11 16:11:33.870340 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.870148    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa435d5-804c-426d-82ea-f238392ff698-plugins-config-volume\") pod \"faa435d5-804c-426d-82ea-f238392ff698\" (UID: \"faa435d5-804c-426d-82ea-f238392ff698\") "
Sep 11 16:11:33.870572 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.870547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa435d5-804c-426d-82ea-f238392ff698-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "faa435d5-804c-426d-82ea-f238392ff698" (UID: "faa435d5-804c-426d-82ea-f238392ff698"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:11:33.872319 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.872229    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa435d5-804c-426d-82ea-f238392ff698-kube-api-access-8f289" (OuterVolumeSpecName: "kube-api-access-8f289") pod "faa435d5-804c-426d-82ea-f238392ff698" (UID: "faa435d5-804c-426d-82ea-f238392ff698"). InnerVolumeSpecName "kube-api-access-8f289". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:11:33.971199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.971157    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f289\" (UniqueName: \"kubernetes.io/projected/faa435d5-804c-426d-82ea-f238392ff698-kube-api-access-8f289\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:11:33.971199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:33.971191    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa435d5-804c-426d-82ea-f238392ff698-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:11:34.128021 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:34.127925    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"]
Sep 11 16:11:34.129789 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:34.129765    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20aaef10-7c7fbd44f5-jgpzk"]
Sep 11 16:11:35.813041 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:35.813001    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:35.813485 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:35.813066    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:35.826367 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:35.826340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:35.838307 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:35.838271    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:11:36.034674 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:36.034640    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faa435d5-804c-426d-82ea-f238392ff698" path="/var/lib/kubelet/pods/faa435d5-804c-426d-82ea-f238392ff698/volumes"
Sep 11 16:11:43.141483 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.141439    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"]
Sep 11 16:11:43.142049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.142027    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa435d5-804c-426d-82ea-f238392ff698" containerName="payload-processing"
Sep 11 16:11:43.142136 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.142052    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa435d5-804c-426d-82ea-f238392ff698" containerName="payload-processing"
Sep 11 16:11:43.142215 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.142202    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa435d5-804c-426d-82ea-f238392ff698" containerName="payload-processing"
Sep 11 16:11:43.146697 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.146675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.150387 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.150362    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-455ccc2b-dockercfg-sx8l7\""
Sep 11 16:11:43.150654 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.150636    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-455ccc2b\""
Sep 11 16:11:43.165597 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.165564    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"]
Sep 11 16:11:43.261681 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.261641    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214dce65-b83e-489e-a900-2ea5ced34aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.261681 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.261689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7vn\" (UniqueName: \"kubernetes.io/projected/214dce65-b83e-489e-a900-2ea5ced34aca-kube-api-access-8r7vn\") pod \"payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.362653 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.362610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214dce65-b83e-489e-a900-2ea5ced34aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.362653 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.362657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7vn\" (UniqueName: \"kubernetes.io/projected/214dce65-b83e-489e-a900-2ea5ced34aca-kube-api-access-8r7vn\") pod \"payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.363400 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.363381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214dce65-b83e-489e-a900-2ea5ced34aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.376095 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.376059    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7vn\" (UniqueName: \"kubernetes.io/projected/214dce65-b83e-489e-a900-2ea5ced34aca-kube-api-access-8r7vn\") pod \"payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.457742 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.457650    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.738141 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:11:43.738111    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214dce65_b83e_489e_a900_2ea5ced34aca.slice/crio-de21a820e993fa1602b1202c8794ebb63f13165cdb4ea8c2095c0906af5d88b6 WatchSource:0}: Error finding container de21a820e993fa1602b1202c8794ebb63f13165cdb4ea8c2095c0906af5d88b6: Status 404 returned error can't find the container with id de21a820e993fa1602b1202c8794ebb63f13165cdb4ea8c2095c0906af5d88b6
Sep 11 16:11:43.739167 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.739141    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"]
Sep 11 16:11:43.858872 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.858838    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp" event={"ID":"214dce65-b83e-489e-a900-2ea5ced34aca","Type":"ContainerStarted","Data":"5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9"}
Sep 11 16:11:43.859096 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.858892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp" event={"ID":"214dce65-b83e-489e-a900-2ea5ced34aca","Type":"ContainerStarted","Data":"de21a820e993fa1602b1202c8794ebb63f13165cdb4ea8c2095c0906af5d88b6"}
Sep 11 16:11:43.859096 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.859023    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:11:43.889626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:43.889564    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp" podStartSLOduration=0.88954358 podStartE2EDuration="889.54358ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:43.886279741 +0000 UTC m=+1000.310097462" watchObservedRunningTime="2026-09-11 16:11:43.88954358 +0000 UTC m=+1000.313361277"
Sep 11 16:11:53.134151 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.134119    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"]
Sep 11 16:11:53.137870 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.137852    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.148170 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.148134    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9c5b09\"/\"default-dockercfg-pvfwt\""
Sep 11 16:11:53.148357 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.148182    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9c5b09\"/\"kube-root-ca.crt\""
Sep 11 16:11:53.148634 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.148406    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9c5b09\"/\"openshift-service-ca.crt\""
Sep 11 16:11:53.148685 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.148489    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9c5b09\"/\"autores-model-9c5b09-kserve-self-signed-certs\""
Sep 11 16:11:53.156708 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.156681    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"]
Sep 11 16:11:53.161696 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.161670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-dshm\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.161838 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.161705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr55\" (UniqueName: \"kubernetes.io/projected/6894a719-7098-4a61-be8b-f0fc7d474bd2-kube-api-access-lnr55\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.161838 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.161736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-model-cache\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.161838 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.161822    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-home\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.161998 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.161863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-tmp-dir\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.161998 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.161927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6894a719-7098-4a61-be8b-f0fc7d474bd2-tls-certs\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.262788 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.262751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-model-cache\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.262788 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.262793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-home\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.263030 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.262814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-tmp-dir\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.263030 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.262856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6894a719-7098-4a61-be8b-f0fc7d474bd2-tls-certs\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.263030 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.262954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-dshm\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.263030 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.262979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr55\" (UniqueName: \"kubernetes.io/projected/6894a719-7098-4a61-be8b-f0fc7d474bd2-kube-api-access-lnr55\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.263282 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.263226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-model-cache\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.263370 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.263291    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-home\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.263370 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.263329    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-tmp-dir\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.265374 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.265348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-dshm\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.265569 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.265548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6894a719-7098-4a61-be8b-f0fc7d474bd2-tls-certs\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.272372 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.272349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr55\" (UniqueName: \"kubernetes.io/projected/6894a719-7098-4a61-be8b-f0fc7d474bd2-kube-api-access-lnr55\") pod \"autores-model-9c5b09-kserve-5d897fc8dc-7n4ld\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") " pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.454876 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.454792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:11:53.748419 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.748387    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"]
Sep 11 16:11:53.749612 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:11:53.749584    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6894a719_7098_4a61_be8b_f0fc7d474bd2.slice/crio-7fa9f92d99ebb1cd14b1c19840847eeac72bfe3da3095aed5d72a6f19b4a8b87 WatchSource:0}: Error finding container 7fa9f92d99ebb1cd14b1c19840847eeac72bfe3da3095aed5d72a6f19b4a8b87: Status 404 returned error can't find the container with id 7fa9f92d99ebb1cd14b1c19840847eeac72bfe3da3095aed5d72a6f19b4a8b87
Sep 11 16:11:53.914909 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.914871    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld" event={"ID":"6894a719-7098-4a61-be8b-f0fc7d474bd2","Type":"ContainerStarted","Data":"50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9"}
Sep 11 16:11:53.915083 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.914919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld" event={"ID":"6894a719-7098-4a61-be8b-f0fc7d474bd2","Type":"ContainerStarted","Data":"7fa9f92d99ebb1cd14b1c19840847eeac72bfe3da3095aed5d72a6f19b4a8b87"}
Sep 11 16:11:53.940586 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:53.940531    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld" podStartSLOduration=0.94051628 podStartE2EDuration="940.51628ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:53.940099585 +0000 UTC m=+1010.363917301" watchObservedRunningTime="2026-09-11 16:11:53.94051628 +0000 UTC m=+1010.364333974"
Sep 11 16:11:54.866775 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:11:54.866742    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:12:03.455314 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:03.455271    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:12:03.455823 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:03.455331    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:12:03.467720 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:03.467689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:12:03.979027 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:03.978992    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:12:08.703683 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.703646    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"]
Sep 11 16:12:08.708563 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.708535    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.711673 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.711644    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8e3121\"/\"default-dockercfg-vgkwz\""
Sep 11 16:12:08.711835 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.711652    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8e3121\"/\"openshift-service-ca.crt\""
Sep 11 16:12:08.713111 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.713087    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8e3121\"/\"test-model-8e3121-kserve-self-signed-certs\""
Sep 11 16:12:08.713254 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.713116    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8e3121\"/\"kube-root-ca.crt\""
Sep 11 16:12:08.718747 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.718722    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"]
Sep 11 16:12:08.817216 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.817174    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-model-cache\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.817422 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.817225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tmp-dir\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.817422 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.817288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tls-certs\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.817422 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.817339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-dshm\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.817538 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.817442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64ls\" (UniqueName: \"kubernetes.io/projected/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-kube-api-access-v64ls\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.817538 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.817476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-home\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.918466 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.918432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-dshm\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.918649 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.918493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64ls\" (UniqueName: \"kubernetes.io/projected/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-kube-api-access-v64ls\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.918649 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.918530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-home\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.918649 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.918591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-model-cache\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.918649 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.918626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tmp-dir\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.918811 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.918656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tls-certs\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.919018 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.918994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-home\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.919079 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.919050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-model-cache\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.919079 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.919059    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tmp-dir\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.920873 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.920851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-dshm\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.921134 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.921116    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tls-certs\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:08.928183 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:08.928149    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64ls\" (UniqueName: \"kubernetes.io/projected/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-kube-api-access-v64ls\") pod \"test-model-8e3121-kserve-7f8cd5b5f7-gtn6v\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") " pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:09.019887 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:09.019846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:09.302679 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:09.302640    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"]
Sep 11 16:12:09.998687 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:09.998649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v" event={"ID":"b453b899-5b0e-4cda-8574-c71dfdeb5bcb","Type":"ContainerStarted","Data":"d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373"}
Sep 11 16:12:09.998687 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:09.998692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v" event={"ID":"b453b899-5b0e-4cda-8574-c71dfdeb5bcb","Type":"ContainerStarted","Data":"024315dea695d6305890d55147f37046b3f1808838ce7e7c1dcfc41403389ea4"}
Sep 11 16:12:10.038873 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:10.038820    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v" podStartSLOduration=2.038805194 podStartE2EDuration="2.038805194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:10.035957802 +0000 UTC m=+1026.459775507" watchObservedRunningTime="2026-09-11 16:12:10.038805194 +0000 UTC m=+1026.462622888"
Sep 11 16:12:19.020337 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:19.020294    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:19.020337 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:19.020343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:19.033125 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:19.033095    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:19.056349 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:19.056314    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:12:21.216111 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.216073    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"]
Sep 11 16:12:21.216499 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.216303    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp" podUID="214dce65-b83e-489e-a900-2ea5ced34aca" containerName="payload-pre-processing" containerID="cri-o://5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9" gracePeriod=30
Sep 11 16:12:21.479188 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.479119    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:12:21.550992 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.550956    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r7vn\" (UniqueName: \"kubernetes.io/projected/214dce65-b83e-489e-a900-2ea5ced34aca-kube-api-access-8r7vn\") pod \"214dce65-b83e-489e-a900-2ea5ced34aca\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") "
Sep 11 16:12:21.551177 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.551023    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214dce65-b83e-489e-a900-2ea5ced34aca-plugins-config-volume\") pod \"214dce65-b83e-489e-a900-2ea5ced34aca\" (UID: \"214dce65-b83e-489e-a900-2ea5ced34aca\") "
Sep 11 16:12:21.553596 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.551875    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214dce65-b83e-489e-a900-2ea5ced34aca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "214dce65-b83e-489e-a900-2ea5ced34aca" (UID: "214dce65-b83e-489e-a900-2ea5ced34aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:12:21.558699 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.554211    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214dce65-b83e-489e-a900-2ea5ced34aca-kube-api-access-8r7vn" (OuterVolumeSpecName: "kube-api-access-8r7vn") pod "214dce65-b83e-489e-a900-2ea5ced34aca" (UID: "214dce65-b83e-489e-a900-2ea5ced34aca"). InnerVolumeSpecName "kube-api-access-8r7vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:21.651920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.651875    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7vn\" (UniqueName: \"kubernetes.io/projected/214dce65-b83e-489e-a900-2ea5ced34aca-kube-api-access-8r7vn\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:21.651920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:21.651912    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/214dce65-b83e-489e-a900-2ea5ced34aca-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:22.060622 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.060585    2559 generic.go:356] "Generic (PLEG): container finished" podID="214dce65-b83e-489e-a900-2ea5ced34aca" containerID="5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9" exitCode=0
Sep 11 16:12:22.060780 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.060665    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"
Sep 11 16:12:22.060780 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.060679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp" event={"ID":"214dce65-b83e-489e-a900-2ea5ced34aca","Type":"ContainerDied","Data":"5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9"}
Sep 11 16:12:22.060780 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.060753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp" event={"ID":"214dce65-b83e-489e-a900-2ea5ced34aca","Type":"ContainerDied","Data":"de21a820e993fa1602b1202c8794ebb63f13165cdb4ea8c2095c0906af5d88b6"}
Sep 11 16:12:22.060895 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.060788    2559 scope.go:117] "RemoveContainer" containerID="5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9"
Sep 11 16:12:22.073330 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.073302    2559 scope.go:117] "RemoveContainer" containerID="5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9"
Sep 11 16:12:22.073654 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:12:22.073631    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9\": container with ID starting with 5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9 not found: ID does not exist" containerID="5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9"
Sep 11 16:12:22.073736 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.073668    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9"} err="failed to get container status \"5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9\": rpc error: code = NotFound desc = could not find container \"5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9\": container with ID starting with 5b50fe1a7c3ee3c65b60f71d2724ea9533254133371f9683d4b5f961a819eac9 not found: ID does not exist"
Sep 11 16:12:22.086529 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.086493    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"]
Sep 11 16:12:22.095499 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:22.095467    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-455ccc2b-5fb7d6fd6b-d4hhp"]
Sep 11 16:12:24.035368 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:24.035227    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214dce65-b83e-489e-a900-2ea5ced34aca" path="/var/lib/kubelet/pods/214dce65-b83e-489e-a900-2ea5ced34aca/volumes"
Sep 11 16:12:25.412970 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:25.412936    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"]
Sep 11 16:12:35.617694 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.617662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"]
Sep 11 16:12:35.618271 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.617859    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm" podUID="3cb20e12-852b-4b71-87c9-25aef938bbe1" containerName="payload-processing" containerID="cri-o://c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576" gracePeriod=30
Sep 11 16:12:35.628377 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.628346    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"]
Sep 11 16:12:35.628562 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.628538    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2" podUID="45921c6b-aa86-491a-844b-acf0f0e9ee0d" containerName="payload-pre-processing" containerID="cri-o://7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5" gracePeriod=30
Sep 11 16:12:35.885948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.885919    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:12:35.906435 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.906400    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:12:35.987675 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.987635    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjhhh\" (UniqueName: \"kubernetes.io/projected/45921c6b-aa86-491a-844b-acf0f0e9ee0d-kube-api-access-rjhhh\") pod \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") "
Sep 11 16:12:35.987848 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.987707    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb20e12-852b-4b71-87c9-25aef938bbe1-plugins-config-volume\") pod \"3cb20e12-852b-4b71-87c9-25aef938bbe1\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") "
Sep 11 16:12:35.987848 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.987755    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45921c6b-aa86-491a-844b-acf0f0e9ee0d-plugins-config-volume\") pod \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\" (UID: \"45921c6b-aa86-491a-844b-acf0f0e9ee0d\") "
Sep 11 16:12:35.987848 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.987817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvvfd\" (UniqueName: \"kubernetes.io/projected/3cb20e12-852b-4b71-87c9-25aef938bbe1-kube-api-access-fvvfd\") pod \"3cb20e12-852b-4b71-87c9-25aef938bbe1\" (UID: \"3cb20e12-852b-4b71-87c9-25aef938bbe1\") "
Sep 11 16:12:35.988108 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.988076    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb20e12-852b-4b71-87c9-25aef938bbe1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3cb20e12-852b-4b71-87c9-25aef938bbe1" (UID: "3cb20e12-852b-4b71-87c9-25aef938bbe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:12:35.988225 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.988166    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45921c6b-aa86-491a-844b-acf0f0e9ee0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45921c6b-aa86-491a-844b-acf0f0e9ee0d" (UID: "45921c6b-aa86-491a-844b-acf0f0e9ee0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:12:35.989915 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.989896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb20e12-852b-4b71-87c9-25aef938bbe1-kube-api-access-fvvfd" (OuterVolumeSpecName: "kube-api-access-fvvfd") pod "3cb20e12-852b-4b71-87c9-25aef938bbe1" (UID: "3cb20e12-852b-4b71-87c9-25aef938bbe1"). InnerVolumeSpecName "kube-api-access-fvvfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:35.989973 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:35.989938    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45921c6b-aa86-491a-844b-acf0f0e9ee0d-kube-api-access-rjhhh" (OuterVolumeSpecName: "kube-api-access-rjhhh") pod "45921c6b-aa86-491a-844b-acf0f0e9ee0d" (UID: "45921c6b-aa86-491a-844b-acf0f0e9ee0d"). InnerVolumeSpecName "kube-api-access-rjhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:36.089354 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.089320    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45921c6b-aa86-491a-844b-acf0f0e9ee0d-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.089354 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.089350    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvvfd\" (UniqueName: \"kubernetes.io/projected/3cb20e12-852b-4b71-87c9-25aef938bbe1-kube-api-access-fvvfd\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.089354 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.089359    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjhhh\" (UniqueName: \"kubernetes.io/projected/45921c6b-aa86-491a-844b-acf0f0e9ee0d-kube-api-access-rjhhh\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.089608 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.089373    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb20e12-852b-4b71-87c9-25aef938bbe1-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.134589 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.134511    2559 generic.go:356] "Generic (PLEG): container finished" podID="3cb20e12-852b-4b71-87c9-25aef938bbe1" containerID="c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576" exitCode=0
Sep 11 16:12:36.134589 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.134560    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"
Sep 11 16:12:36.134763 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.134596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm" event={"ID":"3cb20e12-852b-4b71-87c9-25aef938bbe1","Type":"ContainerDied","Data":"c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576"}
Sep 11 16:12:36.134763 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.134650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm" event={"ID":"3cb20e12-852b-4b71-87c9-25aef938bbe1","Type":"ContainerDied","Data":"422d5801e0a3977e4cb42083d4a2a5d1dd2f3ed3de2225834377986903b250a3"}
Sep 11 16:12:36.134763 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.134674    2559 scope.go:117] "RemoveContainer" containerID="c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576"
Sep 11 16:12:36.135979 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.135957    2559 generic.go:356] "Generic (PLEG): container finished" podID="45921c6b-aa86-491a-844b-acf0f0e9ee0d" containerID="7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5" exitCode=0
Sep 11 16:12:36.136067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.136008    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"
Sep 11 16:12:36.136114 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.136079    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2" event={"ID":"45921c6b-aa86-491a-844b-acf0f0e9ee0d","Type":"ContainerDied","Data":"7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5"}
Sep 11 16:12:36.136160 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.136146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2" event={"ID":"45921c6b-aa86-491a-844b-acf0f0e9ee0d","Type":"ContainerDied","Data":"0ef5e1b2180a97884598089679b994e1d6f74823367cafd12b2fe08343446626"}
Sep 11 16:12:36.143601 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.143582    2559 scope.go:117] "RemoveContainer" containerID="c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576"
Sep 11 16:12:36.143890 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:12:36.143872    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576\": container with ID starting with c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576 not found: ID does not exist" containerID="c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576"
Sep 11 16:12:36.143983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.143899    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576"} err="failed to get container status \"c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576\": rpc error: code = NotFound desc = could not find container \"c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576\": container with ID starting with c4a53a2bdfc293662a63a37dad751bfe1c5e0834b0701973f148c3e296285576 not found: ID does not exist"
Sep 11 16:12:36.143983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.143915    2559 scope.go:117] "RemoveContainer" containerID="7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5"
Sep 11 16:12:36.151845 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.151825    2559 scope.go:117] "RemoveContainer" containerID="7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5"
Sep 11 16:12:36.152114 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:12:36.152097    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5\": container with ID starting with 7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5 not found: ID does not exist" containerID="7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5"
Sep 11 16:12:36.152199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.152122    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5"} err="failed to get container status \"7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5\": rpc error: code = NotFound desc = could not find container \"7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5\": container with ID starting with 7865d0859485953e218a01ceceb453ecd8ba1fbef5b632107ebd085e1e3bb0c5 not found: ID does not exist"
Sep 11 16:12:36.156425 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.156405    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"]
Sep 11 16:12:36.162410 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.162387    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9c5b09-7748fdd8c7-xgthm"]
Sep 11 16:12:36.175550 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.175525    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"]
Sep 11 16:12:36.179941 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:36.179919    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9c5b09-5594dbf766-fngr2"]
Sep 11 16:12:38.034565 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:38.034531    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb20e12-852b-4b71-87c9-25aef938bbe1" path="/var/lib/kubelet/pods/3cb20e12-852b-4b71-87c9-25aef938bbe1/volumes"
Sep 11 16:12:38.034933 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:38.034904    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45921c6b-aa86-491a-844b-acf0f0e9ee0d" path="/var/lib/kubelet/pods/45921c6b-aa86-491a-844b-acf0f0e9ee0d/volumes"
Sep 11 16:12:40.432257 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.432166    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld" podUID="6894a719-7098-4a61-be8b-f0fc7d474bd2" containerName="main" containerID="cri-o://50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9" gracePeriod=45
Sep 11 16:12:40.687275 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.687185    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:12:40.832820 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.832786    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-tmp-dir\") pod \"6894a719-7098-4a61-be8b-f0fc7d474bd2\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") "
Sep 11 16:12:40.833023 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.832864    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6894a719-7098-4a61-be8b-f0fc7d474bd2-tls-certs\") pod \"6894a719-7098-4a61-be8b-f0fc7d474bd2\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") "
Sep 11 16:12:40.833023 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.832894    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-dshm\") pod \"6894a719-7098-4a61-be8b-f0fc7d474bd2\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") "
Sep 11 16:12:40.833023 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.832971    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-home\") pod \"6894a719-7098-4a61-be8b-f0fc7d474bd2\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") "
Sep 11 16:12:40.833023 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833000    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnr55\" (UniqueName: \"kubernetes.io/projected/6894a719-7098-4a61-be8b-f0fc7d474bd2-kube-api-access-lnr55\") pod \"6894a719-7098-4a61-be8b-f0fc7d474bd2\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") "
Sep 11 16:12:40.833219 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833024    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-model-cache\") pod \"6894a719-7098-4a61-be8b-f0fc7d474bd2\" (UID: \"6894a719-7098-4a61-be8b-f0fc7d474bd2\") "
Sep 11 16:12:40.833219 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833070    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6894a719-7098-4a61-be8b-f0fc7d474bd2" (UID: "6894a719-7098-4a61-be8b-f0fc7d474bd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:40.833341 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833312    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6894a719-7098-4a61-be8b-f0fc7d474bd2" (UID: "6894a719-7098-4a61-be8b-f0fc7d474bd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:40.833405 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833349    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-home" (OuterVolumeSpecName: "home") pod "6894a719-7098-4a61-be8b-f0fc7d474bd2" (UID: "6894a719-7098-4a61-be8b-f0fc7d474bd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:40.833469 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833423    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:40.833469 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833437    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:40.833469 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.833445    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:40.835350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.835310    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-dshm" (OuterVolumeSpecName: "dshm") pod "6894a719-7098-4a61-be8b-f0fc7d474bd2" (UID: "6894a719-7098-4a61-be8b-f0fc7d474bd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:40.835350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.835346    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6894a719-7098-4a61-be8b-f0fc7d474bd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6894a719-7098-4a61-be8b-f0fc7d474bd2" (UID: "6894a719-7098-4a61-be8b-f0fc7d474bd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:12:40.835350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.835322    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6894a719-7098-4a61-be8b-f0fc7d474bd2-kube-api-access-lnr55" (OuterVolumeSpecName: "kube-api-access-lnr55") pod "6894a719-7098-4a61-be8b-f0fc7d474bd2" (UID: "6894a719-7098-4a61-be8b-f0fc7d474bd2"). InnerVolumeSpecName "kube-api-access-lnr55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:40.934318 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.934272    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6894a719-7098-4a61-be8b-f0fc7d474bd2-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:40.934318 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.934305    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6894a719-7098-4a61-be8b-f0fc7d474bd2-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:40.934318 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:40.934318    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnr55\" (UniqueName: \"kubernetes.io/projected/6894a719-7098-4a61-be8b-f0fc7d474bd2-kube-api-access-lnr55\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:12:41.164971 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.164936    2559 generic.go:356] "Generic (PLEG): container finished" podID="6894a719-7098-4a61-be8b-f0fc7d474bd2" containerID="50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9" exitCode=0
Sep 11 16:12:41.165139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.164978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld" event={"ID":"6894a719-7098-4a61-be8b-f0fc7d474bd2","Type":"ContainerDied","Data":"50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9"}
Sep 11 16:12:41.165139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.164983    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"
Sep 11 16:12:41.165139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.165008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld" event={"ID":"6894a719-7098-4a61-be8b-f0fc7d474bd2","Type":"ContainerDied","Data":"7fa9f92d99ebb1cd14b1c19840847eeac72bfe3da3095aed5d72a6f19b4a8b87"}
Sep 11 16:12:41.165139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.165024    2559 scope.go:117] "RemoveContainer" containerID="50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9"
Sep 11 16:12:41.174553 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.174534    2559 scope.go:117] "RemoveContainer" containerID="50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9"
Sep 11 16:12:41.174861 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:12:41.174839    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9\": container with ID starting with 50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9 not found: ID does not exist" containerID="50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9"
Sep 11 16:12:41.174943 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.174875    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9"} err="failed to get container status \"50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9\": rpc error: code = NotFound desc = could not find container \"50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9\": container with ID starting with 50c111441148bb709b268edbc2de746c7e44dfc06eb21c977beec6a63f51f4b9 not found: ID does not exist"
Sep 11 16:12:41.193548 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.193512    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"]
Sep 11 16:12:41.200827 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:41.200798    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-9c5b09/autores-model-9c5b09-kserve-5d897fc8dc-7n4ld"]
Sep 11 16:12:42.034960 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:42.034878    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6894a719-7098-4a61-be8b-f0fc7d474bd2" path="/var/lib/kubelet/pods/6894a719-7098-4a61-be8b-f0fc7d474bd2/volumes"
Sep 11 16:12:48.849117 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:48.849078    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"]
Sep 11 16:12:51.080667 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.080630    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"]
Sep 11 16:12:51.081034 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081019    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6894a719-7098-4a61-be8b-f0fc7d474bd2" containerName="main"
Sep 11 16:12:51.081034 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081029    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6894a719-7098-4a61-be8b-f0fc7d474bd2" containerName="main"
Sep 11 16:12:51.081131 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081042    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb20e12-852b-4b71-87c9-25aef938bbe1" containerName="payload-processing"
Sep 11 16:12:51.081131 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081049    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb20e12-852b-4b71-87c9-25aef938bbe1" containerName="payload-processing"
Sep 11 16:12:51.081131 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081070    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214dce65-b83e-489e-a900-2ea5ced34aca" containerName="payload-pre-processing"
Sep 11 16:12:51.081131 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081075    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="214dce65-b83e-489e-a900-2ea5ced34aca" containerName="payload-pre-processing"
Sep 11 16:12:51.081131 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081087    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45921c6b-aa86-491a-844b-acf0f0e9ee0d" containerName="payload-pre-processing"
Sep 11 16:12:51.081131 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081092    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45921c6b-aa86-491a-844b-acf0f0e9ee0d" containerName="payload-pre-processing"
Sep 11 16:12:51.081334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081150    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6894a719-7098-4a61-be8b-f0fc7d474bd2" containerName="main"
Sep 11 16:12:51.081334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081161    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45921c6b-aa86-491a-844b-acf0f0e9ee0d" containerName="payload-pre-processing"
Sep 11 16:12:51.081334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081167    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb20e12-852b-4b71-87c9-25aef938bbe1" containerName="payload-processing"
Sep 11 16:12:51.081334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.081175    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="214dce65-b83e-489e-a900-2ea5ced34aca" containerName="payload-pre-processing"
Sep 11 16:12:51.085913 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.085891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.092468 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.092441    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-dacdbe-dockercfg-hxh4g\""
Sep 11 16:12:51.092575 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.092544    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-dacdbe\""
Sep 11 16:12:51.102478 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.102453    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"]
Sep 11 16:12:51.242756 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.242720    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4611b4f1-df66-4702-804d-1c36a0898600-plugins-config-volume\") pod \"payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") " pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.242930 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.242841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xn7\" (UniqueName: \"kubernetes.io/projected/4611b4f1-df66-4702-804d-1c36a0898600-kube-api-access-q7xn7\") pod \"payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") " pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.344529 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.344435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4611b4f1-df66-4702-804d-1c36a0898600-plugins-config-volume\") pod \"payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") " pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.344685 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.344541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xn7\" (UniqueName: \"kubernetes.io/projected/4611b4f1-df66-4702-804d-1c36a0898600-kube-api-access-q7xn7\") pod \"payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") " pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.345303 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.345281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4611b4f1-df66-4702-804d-1c36a0898600-plugins-config-volume\") pod \"payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") " pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.354363 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.354333    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xn7\" (UniqueName: \"kubernetes.io/projected/4611b4f1-df66-4702-804d-1c36a0898600-kube-api-access-q7xn7\") pod \"payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") " pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.396142 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.396104    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:51.677818 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:51.677796    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"]
Sep 11 16:12:51.679880 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:12:51.679854    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4611b4f1_df66_4702_804d_1c36a0898600.slice/crio-06cdd44493f11dd29b5d3105f1c4bfe60c7c09109a6316ada80216c93e434141 WatchSource:0}: Error finding container 06cdd44493f11dd29b5d3105f1c4bfe60c7c09109a6316ada80216c93e434141: Status 404 returned error can't find the container with id 06cdd44493f11dd29b5d3105f1c4bfe60c7c09109a6316ada80216c93e434141
Sep 11 16:12:52.231316 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:52.231272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g" event={"ID":"4611b4f1-df66-4702-804d-1c36a0898600","Type":"ContainerStarted","Data":"4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15"}
Sep 11 16:12:52.231789 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:52.231343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:12:52.231789 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:52.231366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g" event={"ID":"4611b4f1-df66-4702-804d-1c36a0898600","Type":"ContainerStarted","Data":"06cdd44493f11dd29b5d3105f1c4bfe60c7c09109a6316ada80216c93e434141"}
Sep 11 16:12:52.251965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:12:52.251911    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g" podStartSLOduration=1.251895015 podStartE2EDuration="1.251895015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:52.250406071 +0000 UTC m=+1068.674223778" watchObservedRunningTime="2026-09-11 16:12:52.251895015 +0000 UTC m=+1068.675712708"
Sep 11 16:13:03.238911 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:03.238835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:13:03.868452 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:03.868405    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt" podUID="e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" containerName="main" containerID="cri-o://3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835" gracePeriod=45
Sep 11 16:13:04.132911 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.132846    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:13:04.174889 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.174857    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wphxz\" (UniqueName: \"kubernetes.io/projected/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-kube-api-access-wphxz\") pod \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") "
Sep 11 16:13:04.175057 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.174953    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-model-cache\") pod \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") "
Sep 11 16:13:04.175057 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.174971    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-home\") pod \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") "
Sep 11 16:13:04.175057 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175021    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tmp-dir\") pod \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") "
Sep 11 16:13:04.175057 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175044    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-dshm\") pod \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") "
Sep 11 16:13:04.175288 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175069    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tls-certs\") pod \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\" (UID: \"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab\") "
Sep 11 16:13:04.175358 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175292    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-home" (OuterVolumeSpecName: "home") pod "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" (UID: "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:13:04.175358 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175309    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" (UID: "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:13:04.175358 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175327    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" (UID: "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:13:04.175524 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175507    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:04.175575 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175526    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:04.175575 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.175553    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:04.177285 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.177226    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-dshm" (OuterVolumeSpecName: "dshm") pod "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" (UID: "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:13:04.177285 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.177271    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-kube-api-access-wphxz" (OuterVolumeSpecName: "kube-api-access-wphxz") pod "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" (UID: "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab"). InnerVolumeSpecName "kube-api-access-wphxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:13:04.177443 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.177386    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" (UID: "e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:13:04.276581 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.276535    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wphxz\" (UniqueName: \"kubernetes.io/projected/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-kube-api-access-wphxz\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:04.276581 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.276572    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:04.276581 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.276585    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:04.292110 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.292073    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" containerID="3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835" exitCode=0
Sep 11 16:13:04.292294 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.292115    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"
Sep 11 16:13:04.292294 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.292152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt" event={"ID":"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab","Type":"ContainerDied","Data":"3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835"}
Sep 11 16:13:04.292294 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.292190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt" event={"ID":"e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab","Type":"ContainerDied","Data":"b45ecdd81d89d250fb980a81f61d8cc2e30810f46962c927a12218da99bfbdbb"}
Sep 11 16:13:04.292294 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.292218    2559 scope.go:117] "RemoveContainer" containerID="3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835"
Sep 11 16:13:04.301462 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.301443    2559 scope.go:117] "RemoveContainer" containerID="3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835"
Sep 11 16:13:04.302623 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:13:04.302595    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835\": container with ID starting with 3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835 not found: ID does not exist" containerID="3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835"
Sep 11 16:13:04.302733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.302630    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835"} err="failed to get container status \"3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835\": rpc error: code = NotFound desc = could not find container \"3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835\": container with ID starting with 3123e54d751b145f7cb9aa67e0bb15162834140f0257389a3a6f0a5bd177c835 not found: ID does not exist"
Sep 11 16:13:04.318462 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.318431    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"]
Sep 11 16:13:04.325967 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:04.325935    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-497b3375-kserve-9fbf859d5-shdbt"]
Sep 11 16:13:06.034100 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:06.034064    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" path="/var/lib/kubelet/pods/e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab/volumes"
Sep 11 16:13:07.239065 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.239031    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"]
Sep 11 16:13:07.239523 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.239459    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" containerName="main"
Sep 11 16:13:07.239523 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.239475    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" containerName="main"
Sep 11 16:13:07.239605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.239555    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a3577f-8dd1-4b3d-97a0-8aa0fa7ae7ab" containerName="main"
Sep 11 16:13:07.244063 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.244040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.248708 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.248684    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b1bd5\"/\"test-model-2b1bd5-kserve-self-signed-certs\""
Sep 11 16:13:07.248847 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.248826    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b1bd5\"/\"openshift-service-ca.crt\""
Sep 11 16:13:07.248847 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.248833    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b1bd5\"/\"kube-root-ca.crt\""
Sep 11 16:13:07.248945 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.248841    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b1bd5\"/\"default-dockercfg-qzw2m\""
Sep 11 16:13:07.255307 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.255279    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"]
Sep 11 16:13:07.303686 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.303648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxdx\" (UniqueName: \"kubernetes.io/projected/2678b35a-7941-44c7-b062-9bb7d306c0dd-kube-api-access-wcxdx\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.303865 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.303729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-model-cache\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.303865 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.303764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678b35a-7941-44c7-b062-9bb7d306c0dd-tls-certs\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.303865 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.303835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-tmp-dir\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.303986 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.303877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-home\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.303986 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.303899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-dshm\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.404537 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.404503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxdx\" (UniqueName: \"kubernetes.io/projected/2678b35a-7941-44c7-b062-9bb7d306c0dd-kube-api-access-wcxdx\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.404730 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.404560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-model-cache\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.404730 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.404598    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678b35a-7941-44c7-b062-9bb7d306c0dd-tls-certs\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.404730 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.404631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-tmp-dir\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.404730 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.404652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-home\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.404730 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.404679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-dshm\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.405121 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.405091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-tmp-dir\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.405201 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.405134    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-home\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.405201 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.405102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-model-cache\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.407070 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.407052    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-dshm\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.407183 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.407165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678b35a-7941-44c7-b062-9bb7d306c0dd-tls-certs\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.425215 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.425186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxdx\" (UniqueName: \"kubernetes.io/projected/2678b35a-7941-44c7-b062-9bb7d306c0dd-kube-api-access-wcxdx\") pod \"test-model-2b1bd5-kserve-5694c487bf-wp7qj\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") " pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.554453 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.554409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:07.834359 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:07.834329    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"]
Sep 11 16:13:07.835909 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:13:07.835881    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2678b35a_7941_44c7_b062_9bb7d306c0dd.slice/crio-d35941916a5b6398e2e3c6dbf8467c7990be2803645dde0849e30c40ef6a14a2 WatchSource:0}: Error finding container d35941916a5b6398e2e3c6dbf8467c7990be2803645dde0849e30c40ef6a14a2: Status 404 returned error can't find the container with id d35941916a5b6398e2e3c6dbf8467c7990be2803645dde0849e30c40ef6a14a2
Sep 11 16:13:08.315383 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:08.315340    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj" event={"ID":"2678b35a-7941-44c7-b062-9bb7d306c0dd","Type":"ContainerStarted","Data":"b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793"}
Sep 11 16:13:08.315774 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:08.315394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj" event={"ID":"2678b35a-7941-44c7-b062-9bb7d306c0dd","Type":"ContainerStarted","Data":"d35941916a5b6398e2e3c6dbf8467c7990be2803645dde0849e30c40ef6a14a2"}
Sep 11 16:13:08.337344 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:08.337284    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj" podStartSLOduration=1.33726156 podStartE2EDuration="1.33726156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:13:08.336264242 +0000 UTC m=+1084.760081930" watchObservedRunningTime="2026-09-11 16:13:08.33726156 +0000 UTC m=+1084.761079247"
Sep 11 16:13:17.555332 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:17.555287    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:17.555732 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:17.555343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:17.568283 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:17.568223    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:18.375358 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:18.375327    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:13:55.894579 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:55.894541    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"]
Sep 11 16:13:55.895077 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:55.894834    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g" podUID="4611b4f1-df66-4702-804d-1c36a0898600" containerName="payload-processing" containerID="cri-o://4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15" gracePeriod=30
Sep 11 16:13:56.148856 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.148788    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:13:56.264549 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.264509    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4611b4f1-df66-4702-804d-1c36a0898600-plugins-config-volume\") pod \"4611b4f1-df66-4702-804d-1c36a0898600\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") "
Sep 11 16:13:56.264750 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.264586    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7xn7\" (UniqueName: \"kubernetes.io/projected/4611b4f1-df66-4702-804d-1c36a0898600-kube-api-access-q7xn7\") pod \"4611b4f1-df66-4702-804d-1c36a0898600\" (UID: \"4611b4f1-df66-4702-804d-1c36a0898600\") "
Sep 11 16:13:56.265002 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.264967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4611b4f1-df66-4702-804d-1c36a0898600-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4611b4f1-df66-4702-804d-1c36a0898600" (UID: "4611b4f1-df66-4702-804d-1c36a0898600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:13:56.266736 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.266705    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4611b4f1-df66-4702-804d-1c36a0898600-kube-api-access-q7xn7" (OuterVolumeSpecName: "kube-api-access-q7xn7") pod "4611b4f1-df66-4702-804d-1c36a0898600" (UID: "4611b4f1-df66-4702-804d-1c36a0898600"). InnerVolumeSpecName "kube-api-access-q7xn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:13:56.366284 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.366218    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xn7\" (UniqueName: \"kubernetes.io/projected/4611b4f1-df66-4702-804d-1c36a0898600-kube-api-access-q7xn7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.366284 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.366276    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4611b4f1-df66-4702-804d-1c36a0898600-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.560350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.560314    2559 generic.go:356] "Generic (PLEG): container finished" podID="4611b4f1-df66-4702-804d-1c36a0898600" containerID="4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15" exitCode=0
Sep 11 16:13:56.560563 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.560365    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"
Sep 11 16:13:56.560563 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.560399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g" event={"ID":"4611b4f1-df66-4702-804d-1c36a0898600","Type":"ContainerDied","Data":"4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15"}
Sep 11 16:13:56.560563 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.560459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g" event={"ID":"4611b4f1-df66-4702-804d-1c36a0898600","Type":"ContainerDied","Data":"06cdd44493f11dd29b5d3105f1c4bfe60c7c09109a6316ada80216c93e434141"}
Sep 11 16:13:56.560563 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.560482    2559 scope.go:117] "RemoveContainer" containerID="4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15"
Sep 11 16:13:56.571511 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.571489    2559 scope.go:117] "RemoveContainer" containerID="4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15"
Sep 11 16:13:56.572166 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:13:56.572137    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15\": container with ID starting with 4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15 not found: ID does not exist" containerID="4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15"
Sep 11 16:13:56.572321 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.572177    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15"} err="failed to get container status \"4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15\": rpc error: code = NotFound desc = could not find container \"4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15\": container with ID starting with 4d6c4376e15df692733d0ca6027931fdefeb5d375c4087de118ba23b5844bb15 not found: ID does not exist"
Sep 11 16:13:56.587488 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.587454    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"]
Sep 11 16:13:56.594039 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:56.594007    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-dacdbe-74bc64c95c-cw68g"]
Sep 11 16:13:58.035024 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:13:58.034987    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4611b4f1-df66-4702-804d-1c36a0898600" path="/var/lib/kubelet/pods/4611b4f1-df66-4702-804d-1c36a0898600/volumes"
Sep 11 16:14:09.123599 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.123558    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"]
Sep 11 16:14:09.124010 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.123997    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4611b4f1-df66-4702-804d-1c36a0898600" containerName="payload-processing"
Sep 11 16:14:09.124070 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.124012    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4611b4f1-df66-4702-804d-1c36a0898600" containerName="payload-processing"
Sep 11 16:14:09.124114 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.124086    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4611b4f1-df66-4702-804d-1c36a0898600" containerName="payload-processing"
Sep 11 16:14:09.128366 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.128345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.131248 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.131216    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-faa154ea-dockercfg-49qrj\""
Sep 11 16:14:09.131796 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.131780    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-faa154ea\""
Sep 11 16:14:09.154356 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.154320    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"]
Sep 11 16:14:09.294526 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.294484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7ph\" (UniqueName: \"kubernetes.io/projected/7f260805-1510-4a91-a446-1c8b7ce92d9a-kube-api-access-9f7ph\") pod \"payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.294714 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.294613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f260805-1510-4a91-a446-1c8b7ce92d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.395846 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.395744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7ph\" (UniqueName: \"kubernetes.io/projected/7f260805-1510-4a91-a446-1c8b7ce92d9a-kube-api-access-9f7ph\") pod \"payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.395846 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.395816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f260805-1510-4a91-a446-1c8b7ce92d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.396534 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.396516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f260805-1510-4a91-a446-1c8b7ce92d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.405991 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.405961    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7ph\" (UniqueName: \"kubernetes.io/projected/7f260805-1510-4a91-a446-1c8b7ce92d9a-kube-api-access-9f7ph\") pod \"payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.438328 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.438288    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:09.714594 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:09.714567    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"]
Sep 11 16:14:09.717096 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:14:09.717067    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f260805_1510_4a91_a446_1c8b7ce92d9a.slice/crio-c779a47ed7f54a42d5959ba02973041851ae4645c3aaa047c1643e6aa72b37da WatchSource:0}: Error finding container c779a47ed7f54a42d5959ba02973041851ae4645c3aaa047c1643e6aa72b37da: Status 404 returned error can't find the container with id c779a47ed7f54a42d5959ba02973041851ae4645c3aaa047c1643e6aa72b37da
Sep 11 16:14:10.630485 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:10.630441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb" event={"ID":"7f260805-1510-4a91-a446-1c8b7ce92d9a","Type":"ContainerStarted","Data":"63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13"}
Sep 11 16:14:10.630976 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:10.630615    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:10.630976 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:10.630653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb" event={"ID":"7f260805-1510-4a91-a446-1c8b7ce92d9a","Type":"ContainerStarted","Data":"c779a47ed7f54a42d5959ba02973041851ae4645c3aaa047c1643e6aa72b37da"}
Sep 11 16:14:10.651541 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:10.651486    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb" podStartSLOduration=1.651469095 podStartE2EDuration="1.651469095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:14:10.649501552 +0000 UTC m=+1147.073319257" watchObservedRunningTime="2026-09-11 16:14:10.651469095 +0000 UTC m=+1147.075287209"
Sep 11 16:14:21.637786 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:21.637753    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:33.823920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.823836    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"]
Sep 11 16:14:33.827427 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.827407    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:33.832621 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.832598    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3c2e6c\"/\"kube-root-ca.crt\""
Sep 11 16:14:33.834280 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.833975    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3c2e6c\"/\"default-dockercfg-bbmtr\""
Sep 11 16:14:33.834280 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.833975    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3c2e6c\"/\"openshift-service-ca.crt\""
Sep 11 16:14:33.834280 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.834058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3c2e6c\"/\"noten-model-3c2e6c-kserve-self-signed-certs\""
Sep 11 16:14:33.848263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.848214    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"]
Sep 11 16:14:33.925626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.925575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-model-cache\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:33.925626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.925621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tls-certs\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:33.925851 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.925704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-dshm\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:33.925851 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.925755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ssb\" (UniqueName: \"kubernetes.io/projected/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-kube-api-access-r2ssb\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:33.925851 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.925779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tmp-dir\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:33.925851 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:33.925841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-home\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.026983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.026945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-model-cache\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.026983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.026986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tls-certs\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.027214 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.027060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-dshm\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.027214 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.027110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ssb\" (UniqueName: \"kubernetes.io/projected/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-kube-api-access-r2ssb\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.027214 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.027143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tmp-dir\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.027424 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.027214    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-home\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.027508 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.027483    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-model-cache\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.027592 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.027573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tmp-dir\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.027648 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.027594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-home\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.029386 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.029361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-dshm\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.029640 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.029622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tls-certs\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.037367 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.037338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ssb\" (UniqueName: \"kubernetes.io/projected/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-kube-api-access-r2ssb\") pod \"noten-model-3c2e6c-kserve-78c856f57f-n2qj2\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") " pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.137072 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.136981    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:34.421696 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.421607    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"]
Sep 11 16:14:34.425222 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:14:34.425193    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3a84c53_2cbc_4f6f_8d34_088db6cf4051.slice/crio-05b43bc721a3b4002fd1e8fdd64ce5336a62f3f6e895663e44c81d89fd650491 WatchSource:0}: Error finding container 05b43bc721a3b4002fd1e8fdd64ce5336a62f3f6e895663e44c81d89fd650491: Status 404 returned error can't find the container with id 05b43bc721a3b4002fd1e8fdd64ce5336a62f3f6e895663e44c81d89fd650491
Sep 11 16:14:34.749121 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.749082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2" event={"ID":"f3a84c53-2cbc-4f6f-8d34-088db6cf4051","Type":"ContainerStarted","Data":"fde25298c3892dbc572393b40023a9e0f0bb80fc9663d857d01e1101828c684c"}
Sep 11 16:14:34.749121 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.749127    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2" event={"ID":"f3a84c53-2cbc-4f6f-8d34-088db6cf4051","Type":"ContainerStarted","Data":"05b43bc721a3b4002fd1e8fdd64ce5336a62f3f6e895663e44c81d89fd650491"}
Sep 11 16:14:34.774061 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:34.774004    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2" podStartSLOduration=1.773986751 podStartE2EDuration="1.773986751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:14:34.772675535 +0000 UTC m=+1171.196493243" watchObservedRunningTime="2026-09-11 16:14:34.773986751 +0000 UTC m=+1171.197804444"
Sep 11 16:14:36.041689 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:36.041652    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"]
Sep 11 16:14:44.137360 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:44.137316    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:44.137786 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:44.137381    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:44.150499 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:44.150460    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:44.812684 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:44.812649    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:14:49.438947 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.438902    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"]
Sep 11 16:14:49.439394 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.439164    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb" podUID="7f260805-1510-4a91-a446-1c8b7ce92d9a" containerName="payload-pre-processing" containerID="cri-o://63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13" gracePeriod=30
Sep 11 16:14:49.690406 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.690337    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:49.772925 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.772890    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f260805-1510-4a91-a446-1c8b7ce92d9a-plugins-config-volume\") pod \"7f260805-1510-4a91-a446-1c8b7ce92d9a\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") "
Sep 11 16:14:49.772925 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.772939    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f7ph\" (UniqueName: \"kubernetes.io/projected/7f260805-1510-4a91-a446-1c8b7ce92d9a-kube-api-access-9f7ph\") pod \"7f260805-1510-4a91-a446-1c8b7ce92d9a\" (UID: \"7f260805-1510-4a91-a446-1c8b7ce92d9a\") "
Sep 11 16:14:49.773284 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.773258    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f260805-1510-4a91-a446-1c8b7ce92d9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f260805-1510-4a91-a446-1c8b7ce92d9a" (UID: "7f260805-1510-4a91-a446-1c8b7ce92d9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:14:49.775049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.775025    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f260805-1510-4a91-a446-1c8b7ce92d9a-kube-api-access-9f7ph" (OuterVolumeSpecName: "kube-api-access-9f7ph") pod "7f260805-1510-4a91-a446-1c8b7ce92d9a" (UID: "7f260805-1510-4a91-a446-1c8b7ce92d9a"). InnerVolumeSpecName "kube-api-access-9f7ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:14:49.827684 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.827647    2559 generic.go:356] "Generic (PLEG): container finished" podID="7f260805-1510-4a91-a446-1c8b7ce92d9a" containerID="63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13" exitCode=0
Sep 11 16:14:49.827858 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.827703    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"
Sep 11 16:14:49.827858 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.827731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb" event={"ID":"7f260805-1510-4a91-a446-1c8b7ce92d9a","Type":"ContainerDied","Data":"63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13"}
Sep 11 16:14:49.827858 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.827788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb" event={"ID":"7f260805-1510-4a91-a446-1c8b7ce92d9a","Type":"ContainerDied","Data":"c779a47ed7f54a42d5959ba02973041851ae4645c3aaa047c1643e6aa72b37da"}
Sep 11 16:14:49.827858 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.827815    2559 scope.go:117] "RemoveContainer" containerID="63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13"
Sep 11 16:14:49.837265 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.837221    2559 scope.go:117] "RemoveContainer" containerID="63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13"
Sep 11 16:14:49.837557 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:14:49.837537    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13\": container with ID starting with 63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13 not found: ID does not exist" containerID="63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13"
Sep 11 16:14:49.837628 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.837566    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13"} err="failed to get container status \"63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13\": rpc error: code = NotFound desc = could not find container \"63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13\": container with ID starting with 63f0d01d2d863e7c9a73a8caff440bab3fc8899af3816aa04e770916d2d2dc13 not found: ID does not exist"
Sep 11 16:14:49.860299 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.860267    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"]
Sep 11 16:14:49.867147 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.867117    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-faa154ea-595668b7c4-qhtwb"]
Sep 11 16:14:49.873691 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.873667    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f260805-1510-4a91-a446-1c8b7ce92d9a-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:49.873764 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:49.873694    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f7ph\" (UniqueName: \"kubernetes.io/projected/7f260805-1510-4a91-a446-1c8b7ce92d9a-kube-api-access-9f7ph\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:50.035884 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:50.035851    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f260805-1510-4a91-a446-1c8b7ce92d9a" path="/var/lib/kubelet/pods/7f260805-1510-4a91-a446-1c8b7ce92d9a/volumes"
Sep 11 16:14:51.061406 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.061368    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v" podUID="b453b899-5b0e-4cda-8574-c71dfdeb5bcb" containerName="main" containerID="cri-o://d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373" gracePeriod=45
Sep 11 16:14:51.317551 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.317475    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:14:51.388560 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388524    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tmp-dir\") pod \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") "
Sep 11 16:14:51.388767 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388569    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v64ls\" (UniqueName: \"kubernetes.io/projected/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-kube-api-access-v64ls\") pod \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") "
Sep 11 16:14:51.388767 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388628    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-dshm\") pod \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") "
Sep 11 16:14:51.388767 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-model-cache\") pod \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") "
Sep 11 16:14:51.388767 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388692    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-home\") pod \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") "
Sep 11 16:14:51.389001 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tls-certs\") pod \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\" (UID: \"b453b899-5b0e-4cda-8574-c71dfdeb5bcb\") "
Sep 11 16:14:51.389001 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388866    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b453b899-5b0e-4cda-8574-c71dfdeb5bcb" (UID: "b453b899-5b0e-4cda-8574-c71dfdeb5bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:51.389001 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.388961    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b453b899-5b0e-4cda-8574-c71dfdeb5bcb" (UID: "b453b899-5b0e-4cda-8574-c71dfdeb5bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:51.389149 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.389094    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-home" (OuterVolumeSpecName: "home") pod "b453b899-5b0e-4cda-8574-c71dfdeb5bcb" (UID: "b453b899-5b0e-4cda-8574-c71dfdeb5bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:51.389149 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.389119    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:51.389149 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.389142    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:51.391044 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.391024    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-dshm" (OuterVolumeSpecName: "dshm") pod "b453b899-5b0e-4cda-8574-c71dfdeb5bcb" (UID: "b453b899-5b0e-4cda-8574-c71dfdeb5bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:51.391164 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.391141    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b453b899-5b0e-4cda-8574-c71dfdeb5bcb" (UID: "b453b899-5b0e-4cda-8574-c71dfdeb5bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:14:51.391276 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.391259    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-kube-api-access-v64ls" (OuterVolumeSpecName: "kube-api-access-v64ls") pod "b453b899-5b0e-4cda-8574-c71dfdeb5bcb" (UID: "b453b899-5b0e-4cda-8574-c71dfdeb5bcb"). InnerVolumeSpecName "kube-api-access-v64ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:14:51.490463 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.490427    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v64ls\" (UniqueName: \"kubernetes.io/projected/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-kube-api-access-v64ls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:51.490463 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.490461    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:51.490463 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.490473    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:51.490707 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.490482    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b453b899-5b0e-4cda-8574-c71dfdeb5bcb-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:14:51.839338 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.839303    2559 generic.go:356] "Generic (PLEG): container finished" podID="b453b899-5b0e-4cda-8574-c71dfdeb5bcb" containerID="d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373" exitCode=0
Sep 11 16:14:51.839538 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.839367    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"
Sep 11 16:14:51.839538 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.839377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v" event={"ID":"b453b899-5b0e-4cda-8574-c71dfdeb5bcb","Type":"ContainerDied","Data":"d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373"}
Sep 11 16:14:51.839538 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.839429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v" event={"ID":"b453b899-5b0e-4cda-8574-c71dfdeb5bcb","Type":"ContainerDied","Data":"024315dea695d6305890d55147f37046b3f1808838ce7e7c1dcfc41403389ea4"}
Sep 11 16:14:51.839538 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.839458    2559 scope.go:117] "RemoveContainer" containerID="d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373"
Sep 11 16:14:51.848890 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.848870    2559 scope.go:117] "RemoveContainer" containerID="d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373"
Sep 11 16:14:51.849217 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:14:51.849195    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373\": container with ID starting with d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373 not found: ID does not exist" containerID="d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373"
Sep 11 16:14:51.849334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.849224    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373"} err="failed to get container status \"d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373\": rpc error: code = NotFound desc = could not find container \"d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373\": container with ID starting with d2f4604c9c6b5df2851e70182e9ce330152cf24d4f1c79499068c92fd1a49373 not found: ID does not exist"
Sep 11 16:14:51.860878 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.860851    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"]
Sep 11 16:14:51.868572 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:51.868542    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8e3121/test-model-8e3121-kserve-7f8cd5b5f7-gtn6v"]
Sep 11 16:14:52.036130 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:52.036091    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b453b899-5b0e-4cda-8574-c71dfdeb5bcb" path="/var/lib/kubelet/pods/b453b899-5b0e-4cda-8574-c71dfdeb5bcb/volumes"
Sep 11 16:14:55.915682 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:55.915639    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"]
Sep 11 16:14:56.763458 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:14:56.763418    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"]
Sep 11 16:15:00.151486 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.151449    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"]
Sep 11 16:15:00.151864 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.151842    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b453b899-5b0e-4cda-8574-c71dfdeb5bcb" containerName="main"
Sep 11 16:15:00.151864 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.151852    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b453b899-5b0e-4cda-8574-c71dfdeb5bcb" containerName="main"
Sep 11 16:15:00.151947 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.151868    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f260805-1510-4a91-a446-1c8b7ce92d9a" containerName="payload-pre-processing"
Sep 11 16:15:00.151947 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.151874    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f260805-1510-4a91-a446-1c8b7ce92d9a" containerName="payload-pre-processing"
Sep 11 16:15:00.151947 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.151942    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b453b899-5b0e-4cda-8574-c71dfdeb5bcb" containerName="main"
Sep 11 16:15:00.152048 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.151953    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f260805-1510-4a91-a446-1c8b7ce92d9a" containerName="payload-pre-processing"
Sep 11 16:15:00.156288 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.156268    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"
Sep 11 16:15:00.161052 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.161030    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:15:00.165936 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.165911    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"
Sep 11 16:15:00.174110 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.174083    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"]
Sep 11 16:15:00.441398 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.441370    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"]
Sep 11 16:15:00.443600 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:00.443570    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ee7e92_9773_4f49_af9c_f0dbbbdec2d4.slice/crio-f04caca64c7a2d25ea89e592707f56efbfd9bb588faa479307acf84ec9c94fe1 WatchSource:0}: Error finding container f04caca64c7a2d25ea89e592707f56efbfd9bb588faa479307acf84ec9c94fe1: Status 404 returned error can't find the container with id f04caca64c7a2d25ea89e592707f56efbfd9bb588faa479307acf84ec9c94fe1
Sep 11 16:15:00.445565 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.445547    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:15:00.886542 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:00.886505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m" event={"ID":"b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4","Type":"ContainerStarted","Data":"f04caca64c7a2d25ea89e592707f56efbfd9bb588faa479307acf84ec9c94fe1"}
Sep 11 16:15:03.909403 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:03.909315    2559 generic.go:356] "Generic (PLEG): container finished" podID="b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4" containerID="cc44c8aba8a1c716643a933ddd19f37bee33ed5260d26191a61fceefb5bbd26f" exitCode=0
Sep 11 16:15:03.909403 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:03.909367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m" event={"ID":"b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4","Type":"ContainerDied","Data":"cc44c8aba8a1c716643a933ddd19f37bee33ed5260d26191a61fceefb5bbd26f"}
Sep 11 16:15:05.051160 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:05.051125    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"
Sep 11 16:15:05.921721 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:05.921688    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"
Sep 11 16:15:05.921721 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:05.921715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m" event={"ID":"b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4","Type":"ContainerDied","Data":"f04caca64c7a2d25ea89e592707f56efbfd9bb588faa479307acf84ec9c94fe1"}
Sep 11 16:15:05.921952 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:05.921759    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04caca64c7a2d25ea89e592707f56efbfd9bb588faa479307acf84ec9c94fe1"
Sep 11 16:15:06.773410 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:06.773358    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"]
Sep 11 16:15:06.783802 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:06.783769    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"]
Sep 11 16:15:06.784048 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:06.784006    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx" podUID="5a800ac6-982c-486f-9c47-e57a2a34770b" containerName="payload-pre-processing" containerID="cri-o://c670244abcc1bfe22ec4e3b3a789534465ae15bceb2405a939e9bdbea1bc405e" gracePeriod=30
Sep 11 16:15:06.928822 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:06.928787    2559 generic.go:356] "Generic (PLEG): container finished" podID="5a800ac6-982c-486f-9c47-e57a2a34770b" containerID="c670244abcc1bfe22ec4e3b3a789534465ae15bceb2405a939e9bdbea1bc405e" exitCode=0
Sep 11 16:15:06.929013 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:06.928934    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx" event={"ID":"5a800ac6-982c-486f-9c47-e57a2a34770b","Type":"ContainerDied","Data":"c670244abcc1bfe22ec4e3b3a789534465ae15bceb2405a939e9bdbea1bc405e"}
Sep 11 16:15:06.929131 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:06.929021    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f" podUID="88eed963-6f72-4a6f-b7c7-ddb0529d0844" containerName="payload-processing" containerID="cri-o://11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a" gracePeriod=30
Sep 11 16:15:07.052919 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.052889    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:15:07.157002 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.156967    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a800ac6-982c-486f-9c47-e57a2a34770b-plugins-config-volume\") pod \"5a800ac6-982c-486f-9c47-e57a2a34770b\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") "
Sep 11 16:15:07.157191 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.157141    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckt6g\" (UniqueName: \"kubernetes.io/projected/5a800ac6-982c-486f-9c47-e57a2a34770b-kube-api-access-ckt6g\") pod \"5a800ac6-982c-486f-9c47-e57a2a34770b\" (UID: \"5a800ac6-982c-486f-9c47-e57a2a34770b\") "
Sep 11 16:15:07.157485 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.157456    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a800ac6-982c-486f-9c47-e57a2a34770b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a800ac6-982c-486f-9c47-e57a2a34770b" (UID: "5a800ac6-982c-486f-9c47-e57a2a34770b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:15:07.159203 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.159180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a800ac6-982c-486f-9c47-e57a2a34770b-kube-api-access-ckt6g" (OuterVolumeSpecName: "kube-api-access-ckt6g") pod "5a800ac6-982c-486f-9c47-e57a2a34770b" (UID: "5a800ac6-982c-486f-9c47-e57a2a34770b"). InnerVolumeSpecName "kube-api-access-ckt6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:07.173806 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.173780    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:15:07.258355 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.258311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6w54\" (UniqueName: \"kubernetes.io/projected/88eed963-6f72-4a6f-b7c7-ddb0529d0844-kube-api-access-j6w54\") pod \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") "
Sep 11 16:15:07.258562 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.258415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eed963-6f72-4a6f-b7c7-ddb0529d0844-plugins-config-volume\") pod \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\" (UID: \"88eed963-6f72-4a6f-b7c7-ddb0529d0844\") "
Sep 11 16:15:07.258697 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.258680    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckt6g\" (UniqueName: \"kubernetes.io/projected/5a800ac6-982c-486f-9c47-e57a2a34770b-kube-api-access-ckt6g\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.258757 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.258701    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a800ac6-982c-486f-9c47-e57a2a34770b-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.258795 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.258760    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88eed963-6f72-4a6f-b7c7-ddb0529d0844-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88eed963-6f72-4a6f-b7c7-ddb0529d0844" (UID: "88eed963-6f72-4a6f-b7c7-ddb0529d0844"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:15:07.260433 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.260413    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eed963-6f72-4a6f-b7c7-ddb0529d0844-kube-api-access-j6w54" (OuterVolumeSpecName: "kube-api-access-j6w54") pod "88eed963-6f72-4a6f-b7c7-ddb0529d0844" (UID: "88eed963-6f72-4a6f-b7c7-ddb0529d0844"). InnerVolumeSpecName "kube-api-access-j6w54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:07.359868 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.359775    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eed963-6f72-4a6f-b7c7-ddb0529d0844-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.359868 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.359833    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6w54\" (UniqueName: \"kubernetes.io/projected/88eed963-6f72-4a6f-b7c7-ddb0529d0844-kube-api-access-j6w54\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.935179 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.935147    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"
Sep 11 16:15:07.935594 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.935191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx" event={"ID":"5a800ac6-982c-486f-9c47-e57a2a34770b","Type":"ContainerDied","Data":"b4c8cf48fcb41047cf482357a05a035c8f2edb8513425c28772f270894a7445d"}
Sep 11 16:15:07.935594 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.935288    2559 scope.go:117] "RemoveContainer" containerID="c670244abcc1bfe22ec4e3b3a789534465ae15bceb2405a939e9bdbea1bc405e"
Sep 11 16:15:07.936771 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.936590    2559 generic.go:356] "Generic (PLEG): container finished" podID="88eed963-6f72-4a6f-b7c7-ddb0529d0844" containerID="11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a" exitCode=0
Sep 11 16:15:07.936771 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.936626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f" event={"ID":"88eed963-6f72-4a6f-b7c7-ddb0529d0844","Type":"ContainerDied","Data":"11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a"}
Sep 11 16:15:07.936771 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.936672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f" event={"ID":"88eed963-6f72-4a6f-b7c7-ddb0529d0844","Type":"ContainerDied","Data":"2a8615581542872183dba31d8dae1b98335b66a4238607f39c8284781eef7d15"}
Sep 11 16:15:07.936771 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.936636    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"
Sep 11 16:15:07.945351 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.945183    2559 scope.go:117] "RemoveContainer" containerID="11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a"
Sep 11 16:15:07.953521 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.953503    2559 scope.go:117] "RemoveContainer" containerID="11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a"
Sep 11 16:15:07.953802 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:15:07.953785    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a\": container with ID starting with 11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a not found: ID does not exist" containerID="11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a"
Sep 11 16:15:07.953851 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.953811    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a"} err="failed to get container status \"11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a\": rpc error: code = NotFound desc = could not find container \"11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a\": container with ID starting with 11d2aefdfd1f80c3b198112c7fdc3cfc7af1095992bcd63d6ec22d1db9bd006a not found: ID does not exist"
Sep 11 16:15:07.965355 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.965331    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"]
Sep 11 16:15:07.968997 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.968972    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8e3121-6469d45fdb-xxsrx"]
Sep 11 16:15:07.990263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.990218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"]
Sep 11 16:15:07.995288 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:07.995260    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8e3121-6dff8b5b49-98q5f"]
Sep 11 16:15:08.034636 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:08.034600    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a800ac6-982c-486f-9c47-e57a2a34770b" path="/var/lib/kubelet/pods/5a800ac6-982c-486f-9c47-e57a2a34770b/volumes"
Sep 11 16:15:08.034984 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:08.034972    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88eed963-6f72-4a6f-b7c7-ddb0529d0844" path="/var/lib/kubelet/pods/88eed963-6f72-4a6f-b7c7-ddb0529d0844/volumes"
Sep 11 16:15:10.935268 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:10.935213    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj" podUID="2678b35a-7941-44c7-b062-9bb7d306c0dd" containerName="main" containerID="cri-o://b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793" gracePeriod=45
Sep 11 16:15:11.186964 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.186903    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:15:11.295587 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295546    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678b35a-7941-44c7-b062-9bb7d306c0dd-tls-certs\") pod \"2678b35a-7941-44c7-b062-9bb7d306c0dd\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") "
Sep 11 16:15:11.295587 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295585    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-home\") pod \"2678b35a-7941-44c7-b062-9bb7d306c0dd\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") "
Sep 11 16:15:11.295859 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295627    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-model-cache\") pod \"2678b35a-7941-44c7-b062-9bb7d306c0dd\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") "
Sep 11 16:15:11.295859 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295660    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-dshm\") pod \"2678b35a-7941-44c7-b062-9bb7d306c0dd\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") "
Sep 11 16:15:11.295859 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295728    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcxdx\" (UniqueName: \"kubernetes.io/projected/2678b35a-7941-44c7-b062-9bb7d306c0dd-kube-api-access-wcxdx\") pod \"2678b35a-7941-44c7-b062-9bb7d306c0dd\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") "
Sep 11 16:15:11.295859 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-tmp-dir\") pod \"2678b35a-7941-44c7-b062-9bb7d306c0dd\" (UID: \"2678b35a-7941-44c7-b062-9bb7d306c0dd\") "
Sep 11 16:15:11.296064 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295872    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-home" (OuterVolumeSpecName: "home") pod "2678b35a-7941-44c7-b062-9bb7d306c0dd" (UID: "2678b35a-7941-44c7-b062-9bb7d306c0dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:11.296064 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.295983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "2678b35a-7941-44c7-b062-9bb7d306c0dd" (UID: "2678b35a-7941-44c7-b062-9bb7d306c0dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:11.296064 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.296057    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:11.296270 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.296075    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:11.296270 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.296117    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2678b35a-7941-44c7-b062-9bb7d306c0dd" (UID: "2678b35a-7941-44c7-b062-9bb7d306c0dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:11.298070 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.298046    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2678b35a-7941-44c7-b062-9bb7d306c0dd-kube-api-access-wcxdx" (OuterVolumeSpecName: "kube-api-access-wcxdx") pod "2678b35a-7941-44c7-b062-9bb7d306c0dd" (UID: "2678b35a-7941-44c7-b062-9bb7d306c0dd"). InnerVolumeSpecName "kube-api-access-wcxdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:11.298193 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.298071    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-dshm" (OuterVolumeSpecName: "dshm") pod "2678b35a-7941-44c7-b062-9bb7d306c0dd" (UID: "2678b35a-7941-44c7-b062-9bb7d306c0dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:11.298193 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.298088    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2678b35a-7941-44c7-b062-9bb7d306c0dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2678b35a-7941-44c7-b062-9bb7d306c0dd" (UID: "2678b35a-7941-44c7-b062-9bb7d306c0dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:15:11.397120 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.397080    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxdx\" (UniqueName: \"kubernetes.io/projected/2678b35a-7941-44c7-b062-9bb7d306c0dd-kube-api-access-wcxdx\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:11.397120 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.397112    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:11.397120 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.397125    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678b35a-7941-44c7-b062-9bb7d306c0dd-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:11.397120 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.397132    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678b35a-7941-44c7-b062-9bb7d306c0dd-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:11.782612 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.782577    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2" podUID="f3a84c53-2cbc-4f6f-8d34-088db6cf4051" containerName="main" containerID="cri-o://fde25298c3892dbc572393b40023a9e0f0bb80fc9663d857d01e1101828c684c" gracePeriod=45
Sep 11 16:15:11.959616 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.959586    2559 generic.go:356] "Generic (PLEG): container finished" podID="f3a84c53-2cbc-4f6f-8d34-088db6cf4051" containerID="fde25298c3892dbc572393b40023a9e0f0bb80fc9663d857d01e1101828c684c" exitCode=0
Sep 11 16:15:11.960055 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.959659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2" event={"ID":"f3a84c53-2cbc-4f6f-8d34-088db6cf4051","Type":"ContainerDied","Data":"fde25298c3892dbc572393b40023a9e0f0bb80fc9663d857d01e1101828c684c"}
Sep 11 16:15:11.961137 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.961107    2559 generic.go:356] "Generic (PLEG): container finished" podID="2678b35a-7941-44c7-b062-9bb7d306c0dd" containerID="b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793" exitCode=0
Sep 11 16:15:11.961287 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.961163    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"
Sep 11 16:15:11.961287 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.961192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj" event={"ID":"2678b35a-7941-44c7-b062-9bb7d306c0dd","Type":"ContainerDied","Data":"b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793"}
Sep 11 16:15:11.961287 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.961255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj" event={"ID":"2678b35a-7941-44c7-b062-9bb7d306c0dd","Type":"ContainerDied","Data":"d35941916a5b6398e2e3c6dbf8467c7990be2803645dde0849e30c40ef6a14a2"}
Sep 11 16:15:11.961287 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.961280    2559 scope.go:117] "RemoveContainer" containerID="b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793"
Sep 11 16:15:11.980794 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.980452    2559 scope.go:117] "RemoveContainer" containerID="b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793"
Sep 11 16:15:11.982282 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:15:11.981585    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793\": container with ID starting with b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793 not found: ID does not exist" containerID="b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793"
Sep 11 16:15:11.982282 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.981627    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793"} err="failed to get container status \"b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793\": rpc error: code = NotFound desc = could not find container \"b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793\": container with ID starting with b1ce2a01c9717d53104a0b8f4e3ac1fecd2203da2ce5bd2c0b0c678251db1793 not found: ID does not exist"
Sep 11 16:15:11.993915 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.993880    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"]
Sep 11 16:15:11.999611 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:11.999577    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2b1bd5/test-model-2b1bd5-kserve-5694c487bf-wp7qj"]
Sep 11 16:15:12.028970 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.028941    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:15:12.033955 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.033888    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2678b35a-7941-44c7-b062-9bb7d306c0dd" path="/var/lib/kubelet/pods/2678b35a-7941-44c7-b062-9bb7d306c0dd/volumes"
Sep 11 16:15:12.103363 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103323    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tmp-dir\") pod \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") "
Sep 11 16:15:12.103568 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-model-cache\") pod \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") "
Sep 11 16:15:12.103568 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103455    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tls-certs\") pod \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") "
Sep 11 16:15:12.103568 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103488    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-dshm\") pod \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") "
Sep 11 16:15:12.103568 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103560    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2ssb\" (UniqueName: \"kubernetes.io/projected/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-kube-api-access-r2ssb\") pod \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") "
Sep 11 16:15:12.103782 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103584    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-home\") pod \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\" (UID: \"f3a84c53-2cbc-4f6f-8d34-088db6cf4051\") "
Sep 11 16:15:12.103782 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103646    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3a84c53-2cbc-4f6f-8d34-088db6cf4051" (UID: "f3a84c53-2cbc-4f6f-8d34-088db6cf4051"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:12.103782 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103673    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3a84c53-2cbc-4f6f-8d34-088db6cf4051" (UID: "f3a84c53-2cbc-4f6f-8d34-088db6cf4051"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:12.103917 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103883    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:12.103917 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103900    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:12.103917 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.103880    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-home" (OuterVolumeSpecName: "home") pod "f3a84c53-2cbc-4f6f-8d34-088db6cf4051" (UID: "f3a84c53-2cbc-4f6f-8d34-088db6cf4051"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:12.105878 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.105846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-kube-api-access-r2ssb" (OuterVolumeSpecName: "kube-api-access-r2ssb") pod "f3a84c53-2cbc-4f6f-8d34-088db6cf4051" (UID: "f3a84c53-2cbc-4f6f-8d34-088db6cf4051"). InnerVolumeSpecName "kube-api-access-r2ssb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:12.105878 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.105849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3a84c53-2cbc-4f6f-8d34-088db6cf4051" (UID: "f3a84c53-2cbc-4f6f-8d34-088db6cf4051"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:15:12.106061 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.105874    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-dshm" (OuterVolumeSpecName: "dshm") pod "f3a84c53-2cbc-4f6f-8d34-088db6cf4051" (UID: "f3a84c53-2cbc-4f6f-8d34-088db6cf4051"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:15:12.204982 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.204941    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:12.204982 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.204972    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:12.204982 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.204982    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2ssb\" (UniqueName: \"kubernetes.io/projected/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-kube-api-access-r2ssb\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:12.204982 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.204992    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a84c53-2cbc-4f6f-8d34-088db6cf4051-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:12.967728 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.967691    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"
Sep 11 16:15:12.967728 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.967722    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2" event={"ID":"f3a84c53-2cbc-4f6f-8d34-088db6cf4051","Type":"ContainerDied","Data":"05b43bc721a3b4002fd1e8fdd64ce5336a62f3f6e895663e44c81d89fd650491"}
Sep 11 16:15:12.968284 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.967779    2559 scope.go:117] "RemoveContainer" containerID="fde25298c3892dbc572393b40023a9e0f0bb80fc9663d857d01e1101828c684c"
Sep 11 16:15:12.993231 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.993202    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"]
Sep 11 16:15:12.997722 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:12.997695    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-3c2e6c/noten-model-3c2e6c-kserve-78c856f57f-n2qj2"]
Sep 11 16:15:14.034618 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:14.034579    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a84c53-2cbc-4f6f-8d34-088db6cf4051" path="/var/lib/kubelet/pods/f3a84c53-2cbc-4f6f-8d34-088db6cf4051/volumes"
Sep 11 16:15:15.199169 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199128    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"]
Sep 11 16:15:15.199574 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199557    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a800ac6-982c-486f-9c47-e57a2a34770b" containerName="payload-pre-processing"
Sep 11 16:15:15.199625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199576    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a800ac6-982c-486f-9c47-e57a2a34770b" containerName="payload-pre-processing"
Sep 11 16:15:15.199625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199585    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2678b35a-7941-44c7-b062-9bb7d306c0dd" containerName="main"
Sep 11 16:15:15.199625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199590    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2678b35a-7941-44c7-b062-9bb7d306c0dd" containerName="main"
Sep 11 16:15:15.199625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199605    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a84c53-2cbc-4f6f-8d34-088db6cf4051" containerName="main"
Sep 11 16:15:15.199625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199610    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a84c53-2cbc-4f6f-8d34-088db6cf4051" containerName="main"
Sep 11 16:15:15.199625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199626    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4" containerName="cleanup"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199633    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4" containerName="cleanup"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199643    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88eed963-6f72-4a6f-b7c7-ddb0529d0844" containerName="payload-processing"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199649    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="88eed963-6f72-4a6f-b7c7-ddb0529d0844" containerName="payload-processing"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199705    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a800ac6-982c-486f-9c47-e57a2a34770b" containerName="payload-pre-processing"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199715    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2678b35a-7941-44c7-b062-9bb7d306c0dd" containerName="main"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199722    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a84c53-2cbc-4f6f-8d34-088db6cf4051" containerName="main"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199730    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4" containerName="cleanup"
Sep 11 16:15:15.199807 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.199735    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="88eed963-6f72-4a6f-b7c7-ddb0529d0844" containerName="payload-processing"
Sep 11 16:15:15.203860 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.203839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"
Sep 11 16:15:15.206574 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.206553    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:15:15.211668 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.211263    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"]
Sep 11 16:15:15.335049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.335009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba74a76-5071-449c-baff-a00b14ced6ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq\" (UID: \"8ba74a76-5071-449c-baff-a00b14ced6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"
Sep 11 16:15:15.435603 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.435565    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba74a76-5071-449c-baff-a00b14ced6ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq\" (UID: \"8ba74a76-5071-449c-baff-a00b14ced6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"
Sep 11 16:15:15.436244 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.436211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba74a76-5071-449c-baff-a00b14ced6ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq\" (UID: \"8ba74a76-5071-449c-baff-a00b14ced6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"
Sep 11 16:15:15.515560 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.515526    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"
Sep 11 16:15:15.787778 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.787751    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"]
Sep 11 16:15:15.789654 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:15.789626    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba74a76_5071_449c_baff_a00b14ced6ee.slice/crio-0346902fa11d911a6608429f0e82a11351824ee68c062fb89a8b3de49509c510 WatchSource:0}: Error finding container 0346902fa11d911a6608429f0e82a11351824ee68c062fb89a8b3de49509c510: Status 404 returned error can't find the container with id 0346902fa11d911a6608429f0e82a11351824ee68c062fb89a8b3de49509c510
Sep 11 16:15:15.986031 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:15.985986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq" event={"ID":"8ba74a76-5071-449c-baff-a00b14ced6ee","Type":"ContainerStarted","Data":"0346902fa11d911a6608429f0e82a11351824ee68c062fb89a8b3de49509c510"}
Sep 11 16:15:16.992022 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:16.991925    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ba74a76-5071-449c-baff-a00b14ced6ee" containerID="286ff6c03b53c937dacd10f5caa16c68260a1e02f33c5b56f730da2d63b8c8bd" exitCode=0
Sep 11 16:15:16.992447 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:16.992018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq" event={"ID":"8ba74a76-5071-449c-baff-a00b14ced6ee","Type":"ContainerDied","Data":"286ff6c03b53c937dacd10f5caa16c68260a1e02f33c5b56f730da2d63b8c8bd"}
Sep 11 16:15:18.126542 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:18.126514    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"
Sep 11 16:15:18.266280 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:18.266185    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba74a76-5071-449c-baff-a00b14ced6ee-maas-api-service-ca\") pod \"8ba74a76-5071-449c-baff-a00b14ced6ee\" (UID: \"8ba74a76-5071-449c-baff-a00b14ced6ee\") "
Sep 11 16:15:18.266612 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:18.266579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba74a76-5071-449c-baff-a00b14ced6ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ba74a76-5071-449c-baff-a00b14ced6ee" (UID: "8ba74a76-5071-449c-baff-a00b14ced6ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:15:18.367856 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:18.367814    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba74a76-5071-449c-baff-a00b14ced6ee-maas-api-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:19.001838 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:19.001809    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"
Sep 11 16:15:19.001838 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:19.001824    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq" event={"ID":"8ba74a76-5071-449c-baff-a00b14ced6ee","Type":"ContainerDied","Data":"0346902fa11d911a6608429f0e82a11351824ee68c062fb89a8b3de49509c510"}
Sep 11 16:15:19.002075 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:19.001867    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0346902fa11d911a6608429f0e82a11351824ee68c062fb89a8b3de49509c510"
Sep 11 16:15:20.912601 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.912568    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"]
Sep 11 16:15:20.912961 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.912951    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba74a76-5071-449c-baff-a00b14ced6ee" containerName="revoke-keys"
Sep 11 16:15:20.912961 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.912962    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba74a76-5071-449c-baff-a00b14ced6ee" containerName="revoke-keys"
Sep 11 16:15:20.913049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.913029    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba74a76-5071-449c-baff-a00b14ced6ee" containerName="revoke-keys"
Sep 11 16:15:20.916260 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.916208    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:20.919428 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.919404    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f7c21d\""
Sep 11 16:15:20.938458 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.938427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"]
Sep 11 16:15:20.962592 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.962558    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"]
Sep 11 16:15:20.966177 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.966152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:20.969579 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.969553    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f7c21d-dockercfg-f6s87\""
Sep 11 16:15:20.992812 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.992774    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"]
Sep 11 16:15:20.993068 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.993046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:20.993120 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:20.993084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cql6\" (UniqueName: \"kubernetes.io/projected/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-kube-api-access-6cql6\") pod \"payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:21.093860 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.093814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:21.093860 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.093862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cql6\" (UniqueName: \"kubernetes.io/projected/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-kube-api-access-6cql6\") pod \"payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:21.094086 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.093911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqzp\" (UniqueName: \"kubernetes.io/projected/a143c913-c93b-459d-bf1b-1cce98e8323f-kube-api-access-gmqzp\") pod \"payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:21.094086 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.093931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a143c913-c93b-459d-bf1b-1cce98e8323f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:21.094615 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.094592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:21.104888 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.104861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cql6\" (UniqueName: \"kubernetes.io/projected/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-kube-api-access-6cql6\") pod \"payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:21.194784 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.194678    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqzp\" (UniqueName: \"kubernetes.io/projected/a143c913-c93b-459d-bf1b-1cce98e8323f-kube-api-access-gmqzp\") pod \"payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:21.194784 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.194724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a143c913-c93b-459d-bf1b-1cce98e8323f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:21.195497 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.195477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a143c913-c93b-459d-bf1b-1cce98e8323f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:21.209393 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.209371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqzp\" (UniqueName: \"kubernetes.io/projected/a143c913-c93b-459d-bf1b-1cce98e8323f-kube-api-access-gmqzp\") pod \"payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:21.225921 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.225877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:21.277577 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.277260    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:21.501453 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.501422    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"]
Sep 11 16:15:21.502928 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:21.502899    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b26f06c_21b5_4ed1_a2a6_48cde3ad23bf.slice/crio-9548b727539f8478d6983c97d64afaa26109b764b58d159ef982ab944caad063 WatchSource:0}: Error finding container 9548b727539f8478d6983c97d64afaa26109b764b58d159ef982ab944caad063: Status 404 returned error can't find the container with id 9548b727539f8478d6983c97d64afaa26109b764b58d159ef982ab944caad063
Sep 11 16:15:21.562547 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:21.562519    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"]
Sep 11 16:15:21.564620 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:21.564593    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda143c913_c93b_459d_bf1b_1cce98e8323f.slice/crio-26f2e588b0ae76b760910303bd9839685b3ceebc0d41190b07f9db41c4c1fb91 WatchSource:0}: Error finding container 26f2e588b0ae76b760910303bd9839685b3ceebc0d41190b07f9db41c4c1fb91: Status 404 returned error can't find the container with id 26f2e588b0ae76b760910303bd9839685b3ceebc0d41190b07f9db41c4c1fb91
Sep 11 16:15:22.025346 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.025285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8" event={"ID":"a143c913-c93b-459d-bf1b-1cce98e8323f","Type":"ContainerStarted","Data":"80e3fde2b47262af6df81aaaf4eb590b3a9af11839d0bfcf32ccef93288d2ff8"}
Sep 11 16:15:22.025845 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.025400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8" event={"ID":"a143c913-c93b-459d-bf1b-1cce98e8323f","Type":"ContainerStarted","Data":"26f2e588b0ae76b760910303bd9839685b3ceebc0d41190b07f9db41c4c1fb91"}
Sep 11 16:15:22.025845 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.025436    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:22.027047 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.027024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4" event={"ID":"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf","Type":"ContainerStarted","Data":"2c7f9bcbda46bcf0f131b30bdb34ad6403b1705a2601e886c6540db3c0dfe10e"}
Sep 11 16:15:22.027129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.027065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4" event={"ID":"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf","Type":"ContainerStarted","Data":"9548b727539f8478d6983c97d64afaa26109b764b58d159ef982ab944caad063"}
Sep 11 16:15:22.027129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.027096    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:22.061360 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.061301    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8" podStartSLOduration=2.061284887 podStartE2EDuration="2.061284887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:22.060494463 +0000 UTC m=+1218.484312177" watchObservedRunningTime="2026-09-11 16:15:22.061284887 +0000 UTC m=+1218.485102582"
Sep 11 16:15:22.090973 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:22.090912    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4" podStartSLOduration=2.090893774 podStartE2EDuration="2.090893774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:22.088727926 +0000 UTC m=+1218.512545640" watchObservedRunningTime="2026-09-11 16:15:22.090893774 +0000 UTC m=+1218.514711467"
Sep 11 16:15:25.320017 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.319979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"]
Sep 11 16:15:25.322663 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.322640    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2b1bd5-29819055-f675m"]
Sep 11 16:15:25.354990 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.354953    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"]
Sep 11 16:15:25.355269 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.355221    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t" podUID="8e8be3b9-5d82-4058-a012-89c07626db0e" containerName="payload-processing" containerID="cri-o://ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce" gracePeriod=30
Sep 11 16:15:25.367537 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.367502    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"]
Sep 11 16:15:25.369448 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.368299    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2" podUID="ff51ce42-e695-433d-a2d1-eca44abe0f00" containerName="payload-pre-processing" containerID="cri-o://876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718" gracePeriod=30
Sep 11 16:15:25.595602 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.588227    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"]
Sep 11 16:15:25.595602 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.594844    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b1bd5-7bslq"]
Sep 11 16:15:25.644574 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.644526    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:15:25.647262 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.647228    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:15:25.735372 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.735339    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be3b9-5d82-4058-a012-89c07626db0e-plugins-config-volume\") pod \"8e8be3b9-5d82-4058-a012-89c07626db0e\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") "
Sep 11 16:15:25.735520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.735432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2s29\" (UniqueName: \"kubernetes.io/projected/8e8be3b9-5d82-4058-a012-89c07626db0e-kube-api-access-w2s29\") pod \"8e8be3b9-5d82-4058-a012-89c07626db0e\" (UID: \"8e8be3b9-5d82-4058-a012-89c07626db0e\") "
Sep 11 16:15:25.735520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.735465    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff51ce42-e695-433d-a2d1-eca44abe0f00-plugins-config-volume\") pod \"ff51ce42-e695-433d-a2d1-eca44abe0f00\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") "
Sep 11 16:15:25.735520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.735499    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd68w\" (UniqueName: \"kubernetes.io/projected/ff51ce42-e695-433d-a2d1-eca44abe0f00-kube-api-access-dd68w\") pod \"ff51ce42-e695-433d-a2d1-eca44abe0f00\" (UID: \"ff51ce42-e695-433d-a2d1-eca44abe0f00\") "
Sep 11 16:15:25.735831 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.735805    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8be3b9-5d82-4058-a012-89c07626db0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e8be3b9-5d82-4058-a012-89c07626db0e" (UID: "8e8be3b9-5d82-4058-a012-89c07626db0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:15:25.735901 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.735856    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff51ce42-e695-433d-a2d1-eca44abe0f00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff51ce42-e695-433d-a2d1-eca44abe0f00" (UID: "ff51ce42-e695-433d-a2d1-eca44abe0f00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:15:25.737734 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.737708    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8be3b9-5d82-4058-a012-89c07626db0e-kube-api-access-w2s29" (OuterVolumeSpecName: "kube-api-access-w2s29") pod "8e8be3b9-5d82-4058-a012-89c07626db0e" (UID: "8e8be3b9-5d82-4058-a012-89c07626db0e"). InnerVolumeSpecName "kube-api-access-w2s29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:25.737821 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.737714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff51ce42-e695-433d-a2d1-eca44abe0f00-kube-api-access-dd68w" (OuterVolumeSpecName: "kube-api-access-dd68w") pod "ff51ce42-e695-433d-a2d1-eca44abe0f00" (UID: "ff51ce42-e695-433d-a2d1-eca44abe0f00"). InnerVolumeSpecName "kube-api-access-dd68w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:25.836313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.836169    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2s29\" (UniqueName: \"kubernetes.io/projected/8e8be3b9-5d82-4058-a012-89c07626db0e-kube-api-access-w2s29\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:25.836313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.836212    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff51ce42-e695-433d-a2d1-eca44abe0f00-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:25.836313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.836222    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd68w\" (UniqueName: \"kubernetes.io/projected/ff51ce42-e695-433d-a2d1-eca44abe0f00-kube-api-access-dd68w\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:25.836313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:25.836231    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8be3b9-5d82-4058-a012-89c07626db0e-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:15:26.034399 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.034363    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba74a76-5071-449c-baff-a00b14ced6ee" path="/var/lib/kubelet/pods/8ba74a76-5071-449c-baff-a00b14ced6ee/volumes"
Sep 11 16:15:26.034762 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.034742    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4" path="/var/lib/kubelet/pods/b2ee7e92-9773-4f49-af9c-f0dbbbdec2d4/volumes"
Sep 11 16:15:26.048067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.047957    2559 generic.go:356] "Generic (PLEG): container finished" podID="ff51ce42-e695-433d-a2d1-eca44abe0f00" containerID="876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718" exitCode=0
Sep 11 16:15:26.048067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.048023    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"
Sep 11 16:15:26.048292 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.048062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2" event={"ID":"ff51ce42-e695-433d-a2d1-eca44abe0f00","Type":"ContainerDied","Data":"876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718"}
Sep 11 16:15:26.048292 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.048118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2" event={"ID":"ff51ce42-e695-433d-a2d1-eca44abe0f00","Type":"ContainerDied","Data":"ccbc5cb665c4f56f2d743b43430146f3a3572b5d6f7db7984f3528e4d1d508c4"}
Sep 11 16:15:26.048292 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.048152    2559 scope.go:117] "RemoveContainer" containerID="876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718"
Sep 11 16:15:26.049817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.049791    2559 generic.go:356] "Generic (PLEG): container finished" podID="8e8be3b9-5d82-4058-a012-89c07626db0e" containerID="ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce" exitCode=0
Sep 11 16:15:26.049917 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.049832    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"
Sep 11 16:15:26.049977 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.049877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t" event={"ID":"8e8be3b9-5d82-4058-a012-89c07626db0e","Type":"ContainerDied","Data":"ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce"}
Sep 11 16:15:26.049977 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.049961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t" event={"ID":"8e8be3b9-5d82-4058-a012-89c07626db0e","Type":"ContainerDied","Data":"ff8d28faf35f892b5f56e1da9a9454517e84e998fa03cc924e9c2919ee1dcd4f"}
Sep 11 16:15:26.057047 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.057029    2559 scope.go:117] "RemoveContainer" containerID="876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718"
Sep 11 16:15:26.057381 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:15:26.057358    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718\": container with ID starting with 876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718 not found: ID does not exist" containerID="876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718"
Sep 11 16:15:26.057476 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.057392    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718"} err="failed to get container status \"876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718\": rpc error: code = NotFound desc = could not find container \"876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718\": container with ID starting with 876a7bef6c5bccc4998c9638484b06dade05881130e89a20eb191410def98718 not found: ID does not exist"
Sep 11 16:15:26.057476 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.057415    2559 scope.go:117] "RemoveContainer" containerID="ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce"
Sep 11 16:15:26.065441 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.065421    2559 scope.go:117] "RemoveContainer" containerID="ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce"
Sep 11 16:15:26.065738 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:15:26.065714    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce\": container with ID starting with ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce not found: ID does not exist" containerID="ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce"
Sep 11 16:15:26.065812 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.065750    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce"} err="failed to get container status \"ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce\": rpc error: code = NotFound desc = could not find container \"ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce\": container with ID starting with ed310b3637a65930420491b99e799b6ae41417494c262ce7db21ee9f087488ce not found: ID does not exist"
Sep 11 16:15:26.071398 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.071371    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"]
Sep 11 16:15:26.079027 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.078999    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b1bd5-6986467b4b-zbtr2"]
Sep 11 16:15:26.090986 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.090917    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"]
Sep 11 16:15:26.097059 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:26.097034    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b1bd5-57596cbdd7-5dt2t"]
Sep 11 16:15:28.034201 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:28.034170    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8be3b9-5d82-4058-a012-89c07626db0e" path="/var/lib/kubelet/pods/8e8be3b9-5d82-4058-a012-89c07626db0e/volumes"
Sep 11 16:15:28.034597 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:28.034568    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff51ce42-e695-433d-a2d1-eca44abe0f00" path="/var/lib/kubelet/pods/ff51ce42-e695-433d-a2d1-eca44abe0f00/volumes"
Sep 11 16:15:33.034761 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:33.034726    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:15:33.035121 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:33.034794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:15:40.014356 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.014318    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"]
Sep 11 16:15:40.014824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.014722    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8be3b9-5d82-4058-a012-89c07626db0e" containerName="payload-processing"
Sep 11 16:15:40.014824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.014733    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8be3b9-5d82-4058-a012-89c07626db0e" containerName="payload-processing"
Sep 11 16:15:40.014824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.014747    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff51ce42-e695-433d-a2d1-eca44abe0f00" containerName="payload-pre-processing"
Sep 11 16:15:40.014824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.014753    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff51ce42-e695-433d-a2d1-eca44abe0f00" containerName="payload-pre-processing"
Sep 11 16:15:40.014824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.014814    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8be3b9-5d82-4058-a012-89c07626db0e" containerName="payload-processing"
Sep 11 16:15:40.014824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.014825    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff51ce42-e695-433d-a2d1-eca44abe0f00" containerName="payload-pre-processing"
Sep 11 16:15:40.021359 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.021334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.030843 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.030816    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-59f108-dockercfg-kj2hd\""
Sep 11 16:15:40.030987 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.030860    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-59f108\""
Sep 11 16:15:40.045114 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.045082    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"]
Sep 11 16:15:40.060953 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.060923    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"]
Sep 11 16:15:40.064590 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.064565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.083037 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.083005    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"]
Sep 11 16:15:40.168593 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.168531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gxk\" (UniqueName: \"kubernetes.io/projected/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-kube-api-access-84gxk\") pod \"payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.168793 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.168623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmf4\" (UniqueName: \"kubernetes.io/projected/82dff3fe-b23b-4ca4-9552-45f9ca40a930-kube-api-access-8fmf4\") pod \"payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.168793 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.168694    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dff3fe-b23b-4ca4-9552-45f9ca40a930-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.168890 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.168797    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.269836 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.269734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gxk\" (UniqueName: \"kubernetes.io/projected/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-kube-api-access-84gxk\") pod \"payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.269836 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.269791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmf4\" (UniqueName: \"kubernetes.io/projected/82dff3fe-b23b-4ca4-9552-45f9ca40a930-kube-api-access-8fmf4\") pod \"payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.269836 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.269839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dff3fe-b23b-4ca4-9552-45f9ca40a930-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.270088 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.269860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.270622 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.270600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dff3fe-b23b-4ca4-9552-45f9ca40a930-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.270678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.270621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.285776 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.285740    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmf4\" (UniqueName: \"kubernetes.io/projected/82dff3fe-b23b-4ca4-9552-45f9ca40a930-kube-api-access-8fmf4\") pod \"payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.286631 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.286608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gxk\" (UniqueName: \"kubernetes.io/projected/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-kube-api-access-84gxk\") pod \"payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.332651 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.332615    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:40.375369 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.375326    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:40.614699 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.614673    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"]
Sep 11 16:15:40.615861 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:40.615814    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dff037_59a9_44d9_ad0e_fa2c61e3df8b.slice/crio-287d12e7abe7f917d7e45a558ad06932ced984ad865853a6b718a5207806e24e WatchSource:0}: Error finding container 287d12e7abe7f917d7e45a558ad06932ced984ad865853a6b718a5207806e24e: Status 404 returned error can't find the container with id 287d12e7abe7f917d7e45a558ad06932ced984ad865853a6b718a5207806e24e
Sep 11 16:15:40.665137 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:40.665068    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"]
Sep 11 16:15:40.667146 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:40.667117    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dff3fe_b23b_4ca4_9552_45f9ca40a930.slice/crio-095716e896e1aa2ef8d177a1d5479acdf8490a62c388219ab720497715b969e2 WatchSource:0}: Error finding container 095716e896e1aa2ef8d177a1d5479acdf8490a62c388219ab720497715b969e2: Status 404 returned error can't find the container with id 095716e896e1aa2ef8d177a1d5479acdf8490a62c388219ab720497715b969e2
Sep 11 16:15:41.126662 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.126630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw" event={"ID":"82dff3fe-b23b-4ca4-9552-45f9ca40a930","Type":"ContainerStarted","Data":"caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1"}
Sep 11 16:15:41.127160 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.126697    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:41.127160 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.126720    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw" event={"ID":"82dff3fe-b23b-4ca4-9552-45f9ca40a930","Type":"ContainerStarted","Data":"095716e896e1aa2ef8d177a1d5479acdf8490a62c388219ab720497715b969e2"}
Sep 11 16:15:41.128407 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.128378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp" event={"ID":"56dff037-59a9-44d9-ad0e-fa2c61e3df8b","Type":"ContainerStarted","Data":"b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503"}
Sep 11 16:15:41.128541 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.128424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp" event={"ID":"56dff037-59a9-44d9-ad0e-fa2c61e3df8b","Type":"ContainerStarted","Data":"287d12e7abe7f917d7e45a558ad06932ced984ad865853a6b718a5207806e24e"}
Sep 11 16:15:41.128541 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.128445    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:41.172529 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.172472    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw" podStartSLOduration=1.172453755 podStartE2EDuration="1.172453755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:41.169393613 +0000 UTC m=+1237.593211307" watchObservedRunningTime="2026-09-11 16:15:41.172453755 +0000 UTC m=+1237.596271450"
Sep 11 16:15:41.231726 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:41.231666    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp" podStartSLOduration=2.231648451 podStartE2EDuration="2.231648451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:41.229176942 +0000 UTC m=+1237.652994701" watchObservedRunningTime="2026-09-11 16:15:41.231648451 +0000 UTC m=+1237.655466145"
Sep 11 16:15:51.601137 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.601099    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"]
Sep 11 16:15:51.604722 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.604697    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:51.607734 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.607704    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-320b99\""
Sep 11 16:15:51.608100 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.608078    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-320b99-dockercfg-zx8wh\""
Sep 11 16:15:51.647402 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.647369    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"]
Sep 11 16:15:51.675731 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.675699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe37b99-eec2-433f-9913-5159e5acb8b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:51.675941 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.675749    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxq4d\" (UniqueName: \"kubernetes.io/projected/9fe37b99-eec2-433f-9913-5159e5acb8b2-kube-api-access-gxq4d\") pod \"payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:51.722864 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.722824    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"]
Sep 11 16:15:51.726470 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.726449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.741147 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:15:51.741114    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="c2ae7293-78f3-4f12-926e-e7bb546d350b" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.759958 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.759924    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"]
Sep 11 16:15:51.776617 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.776583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsch7\" (UniqueName: \"kubernetes.io/projected/c2ae7293-78f3-4f12-926e-e7bb546d350b-kube-api-access-hsch7\") pod \"payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.776817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.776649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe37b99-eec2-433f-9913-5159e5acb8b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:51.776817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.776715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxq4d\" (UniqueName: \"kubernetes.io/projected/9fe37b99-eec2-433f-9913-5159e5acb8b2-kube-api-access-gxq4d\") pod \"payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:51.776817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.776738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ae7293-78f3-4f12-926e-e7bb546d350b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.777654 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.777625    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe37b99-eec2-433f-9913-5159e5acb8b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:51.791591 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.791561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxq4d\" (UniqueName: \"kubernetes.io/projected/9fe37b99-eec2-433f-9913-5159e5acb8b2-kube-api-access-gxq4d\") pod \"payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:51.877739 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.877655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsch7\" (UniqueName: \"kubernetes.io/projected/c2ae7293-78f3-4f12-926e-e7bb546d350b-kube-api-access-hsch7\") pod \"payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.877739 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.877733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ae7293-78f3-4f12-926e-e7bb546d350b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.878461 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.878441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ae7293-78f3-4f12-926e-e7bb546d350b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.893916 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.893883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsch7\" (UniqueName: \"kubernetes.io/projected/c2ae7293-78f3-4f12-926e-e7bb546d350b-kube-api-access-hsch7\") pod \"payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:51.915906 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:51.915852    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:52.036992 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:52.036965    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:52.137509 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:52.137417    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:15:52.137509 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:52.137498    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:15:52.222588 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:52.222510    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"]
Sep 11 16:15:52.228835 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:52.228784    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe37b99_eec2_433f_9913_5159e5acb8b2.slice/crio-eb6c2c3ee8f80aa7e9c5af09f431d8a4ee6bdfc04bc1aa1cb849309de76ba118 WatchSource:0}: Error finding container eb6c2c3ee8f80aa7e9c5af09f431d8a4ee6bdfc04bc1aa1cb849309de76ba118: Status 404 returned error can't find the container with id eb6c2c3ee8f80aa7e9c5af09f431d8a4ee6bdfc04bc1aa1cb849309de76ba118
Sep 11 16:15:52.317863 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:52.317837    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"]
Sep 11 16:15:52.320826 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:52.320793    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ae7293_78f3_4f12_926e_e7bb546d350b.slice/crio-1a8aab57182b71f75d7cff012cd6f32ef1dd555fb339fa0bd04bcd1ac59cff46 WatchSource:0}: Error finding container 1a8aab57182b71f75d7cff012cd6f32ef1dd555fb339fa0bd04bcd1ac59cff46: Status 404 returned error can't find the container with id 1a8aab57182b71f75d7cff012cd6f32ef1dd555fb339fa0bd04bcd1ac59cff46
Sep 11 16:15:53.194417 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.194374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9" event={"ID":"9fe37b99-eec2-433f-9913-5159e5acb8b2","Type":"ContainerStarted","Data":"1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe"}
Sep 11 16:15:53.194860 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.194579    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:15:53.194860 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.194616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9" event={"ID":"9fe37b99-eec2-433f-9913-5159e5acb8b2","Type":"ContainerStarted","Data":"eb6c2c3ee8f80aa7e9c5af09f431d8a4ee6bdfc04bc1aa1cb849309de76ba118"}
Sep 11 16:15:53.196152 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.196128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr" event={"ID":"c2ae7293-78f3-4f12-926e-e7bb546d350b","Type":"ContainerStarted","Data":"76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9"}
Sep 11 16:15:53.196291 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.196169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr" event={"ID":"c2ae7293-78f3-4f12-926e-e7bb546d350b","Type":"ContainerStarted","Data":"1a8aab57182b71f75d7cff012cd6f32ef1dd555fb339fa0bd04bcd1ac59cff46"}
Sep 11 16:15:53.196356 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.196288    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:15:53.244382 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.244325    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9" podStartSLOduration=2.244308495 podStartE2EDuration="2.244308495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:53.238264021 +0000 UTC m=+1249.662081726" watchObservedRunningTime="2026-09-11 16:15:53.244308495 +0000 UTC m=+1249.668126189"
Sep 11 16:15:53.298488 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:53.298432    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr" podStartSLOduration=2.298415258 podStartE2EDuration="2.298415258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:53.294912601 +0000 UTC m=+1249.718730315" watchObservedRunningTime="2026-09-11 16:15:53.298415258 +0000 UTC m=+1249.722232968"
Sep 11 16:15:58.286501 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.286420    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"]
Sep 11 16:15:58.291171 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.291152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.294165 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.294144    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1e0e69-dockercfg-vxbm7\""
Sep 11 16:15:58.294689 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.294675    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1e0e69\""
Sep 11 16:15:58.305890 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.305858    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"]
Sep 11 16:15:58.338764 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.338723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6fc\" (UniqueName: \"kubernetes.io/projected/a53a2eba-bdd7-44ad-8839-3632ca23166f-kube-api-access-2g6fc\") pod \"payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.338936 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.338838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53a2eba-bdd7-44ad-8839-3632ca23166f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.440201 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.440150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6fc\" (UniqueName: \"kubernetes.io/projected/a53a2eba-bdd7-44ad-8839-3632ca23166f-kube-api-access-2g6fc\") pod \"payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.440449 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.440304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53a2eba-bdd7-44ad-8839-3632ca23166f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.441039 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.441010    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53a2eba-bdd7-44ad-8839-3632ca23166f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.449312 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.449287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6fc\" (UniqueName: \"kubernetes.io/projected/a53a2eba-bdd7-44ad-8839-3632ca23166f-kube-api-access-2g6fc\") pod \"payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.602659 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.602547    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:58.887769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:58.887736    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"]
Sep 11 16:15:58.888760 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:15:58.888734    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53a2eba_bdd7_44ad_8839_3632ca23166f.slice/crio-c1906ae5b41e23f62a19a73bfe683812ec390a66f42842c9853c2f25007d8b80 WatchSource:0}: Error finding container c1906ae5b41e23f62a19a73bfe683812ec390a66f42842c9853c2f25007d8b80: Status 404 returned error can't find the container with id c1906ae5b41e23f62a19a73bfe683812ec390a66f42842c9853c2f25007d8b80
Sep 11 16:15:59.229438 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:59.229347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh" event={"ID":"a53a2eba-bdd7-44ad-8839-3632ca23166f","Type":"ContainerStarted","Data":"1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb"}
Sep 11 16:15:59.229438 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:59.229405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh" event={"ID":"a53a2eba-bdd7-44ad-8839-3632ca23166f","Type":"ContainerStarted","Data":"c1906ae5b41e23f62a19a73bfe683812ec390a66f42842c9853c2f25007d8b80"}
Sep 11 16:15:59.229438 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:59.229430    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:15:59.254542 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:15:59.254485    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh" podStartSLOduration=1.254467656 podStartE2EDuration="1.254467656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:59.254049536 +0000 UTC m=+1255.677867255" watchObservedRunningTime="2026-09-11 16:15:59.254467656 +0000 UTC m=+1255.678285396"
Sep 11 16:16:04.204307 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:04.204275    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:16:04.204770 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:04.204352    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:16:10.237016 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:10.236985    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:16:35.878476 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.878438    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"]
Sep 11 16:16:35.881521 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.881504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:35.884783 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.884764    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9da1d032\""
Sep 11 16:16:35.897457 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.897430    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"]
Sep 11 16:16:35.950802 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.950762    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"]
Sep 11 16:16:35.954438 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.954414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:35.959583 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.959559    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9da1d032-dockercfg-tb26p\""
Sep 11 16:16:35.975611 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.975577    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"]
Sep 11 16:16:35.989629 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.989588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drqr6\" (UniqueName: \"kubernetes.io/projected/57fc771c-d749-4037-9d20-f0c2ba6067bf-kube-api-access-drqr6\") pod \"payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") " pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:35.989817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.989646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc771c-d749-4037-9d20-f0c2ba6067bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") " pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:35.989817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.989737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvf7\" (UniqueName: \"kubernetes.io/projected/db4d59a1-acf7-4676-94bc-6574ede54151-kube-api-access-nvvf7\") pod \"payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:35.989817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:35.989780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d59a1-acf7-4676-94bc-6574ede54151-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:36.090845 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.090799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drqr6\" (UniqueName: \"kubernetes.io/projected/57fc771c-d749-4037-9d20-f0c2ba6067bf-kube-api-access-drqr6\") pod \"payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") " pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:36.090845 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.090847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc771c-d749-4037-9d20-f0c2ba6067bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") " pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:36.091076 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.090895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvf7\" (UniqueName: \"kubernetes.io/projected/db4d59a1-acf7-4676-94bc-6574ede54151-kube-api-access-nvvf7\") pod \"payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:36.091076 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.090921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d59a1-acf7-4676-94bc-6574ede54151-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:36.091702 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.091681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc771c-d749-4037-9d20-f0c2ba6067bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") " pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:36.091746 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.091687    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d59a1-acf7-4676-94bc-6574ede54151-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:36.100478 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.100446    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvf7\" (UniqueName: \"kubernetes.io/projected/db4d59a1-acf7-4676-94bc-6574ede54151-kube-api-access-nvvf7\") pod \"payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:36.100634 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.100515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drqr6\" (UniqueName: \"kubernetes.io/projected/57fc771c-d749-4037-9d20-f0c2ba6067bf-kube-api-access-drqr6\") pod \"payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") " pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:36.191915 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.191815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:36.264609 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.264576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:36.501536 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.501505    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"]
Sep 11 16:16:36.501665 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:16:36.501627    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fc771c_d749_4037_9d20_f0c2ba6067bf.slice/crio-3a0a8e16c08ad20a9cc68470c977598168d1bad22ae85e3c1c2c1ed6a88d3eda WatchSource:0}: Error finding container 3a0a8e16c08ad20a9cc68470c977598168d1bad22ae85e3c1c2c1ed6a88d3eda: Status 404 returned error can't find the container with id 3a0a8e16c08ad20a9cc68470c977598168d1bad22ae85e3c1c2c1ed6a88d3eda
Sep 11 16:16:36.565250 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:36.565210    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"]
Sep 11 16:16:36.566712 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:16:36.566683    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d59a1_acf7_4676_94bc_6574ede54151.slice/crio-d03dc03d5800f8f4ce2f2d20f87001860e9c74db9c7a1b2b28f518e4588df9e5 WatchSource:0}: Error finding container d03dc03d5800f8f4ce2f2d20f87001860e9c74db9c7a1b2b28f518e4588df9e5: Status 404 returned error can't find the container with id d03dc03d5800f8f4ce2f2d20f87001860e9c74db9c7a1b2b28f518e4588df9e5
Sep 11 16:16:37.422151 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.422112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8" event={"ID":"db4d59a1-acf7-4676-94bc-6574ede54151","Type":"ContainerStarted","Data":"46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711"}
Sep 11 16:16:37.422675 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.422217    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:37.422675 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.422268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8" event={"ID":"db4d59a1-acf7-4676-94bc-6574ede54151","Type":"ContainerStarted","Data":"d03dc03d5800f8f4ce2f2d20f87001860e9c74db9c7a1b2b28f518e4588df9e5"}
Sep 11 16:16:37.424039 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.424015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5" event={"ID":"57fc771c-d749-4037-9d20-f0c2ba6067bf","Type":"ContainerStarted","Data":"c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe"}
Sep 11 16:16:37.424177 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.424052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5" event={"ID":"57fc771c-d749-4037-9d20-f0c2ba6067bf","Type":"ContainerStarted","Data":"3a0a8e16c08ad20a9cc68470c977598168d1bad22ae85e3c1c2c1ed6a88d3eda"}
Sep 11 16:16:37.424177 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.424096    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:37.443587 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.443530    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8" podStartSLOduration=2.443512014 podStartE2EDuration="2.443512014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:37.441725122 +0000 UTC m=+1293.865542840" watchObservedRunningTime="2026-09-11 16:16:37.443512014 +0000 UTC m=+1293.867329707"
Sep 11 16:16:37.464610 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:37.464558    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5" podStartSLOduration=2.4645391229999998 podStartE2EDuration="2.464539123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:37.462935882 +0000 UTC m=+1293.886753576" watchObservedRunningTime="2026-09-11 16:16:37.464539123 +0000 UTC m=+1293.888356816"
Sep 11 16:16:46.700703 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.700665    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"]
Sep 11 16:16:46.704705 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.704685    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.708600 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.708576    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"default-dockercfg-9kwxq\""
Sep 11 16:16:46.709773 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.709752    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"auth-test-model-59f108-kserve-self-signed-certs\""
Sep 11 16:16:46.709892 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.709819    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"openshift-service-ca.crt\""
Sep 11 16:16:46.709892 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.709822    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"kube-root-ca.crt\""
Sep 11 16:16:46.734477 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.734438    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"]
Sep 11 16:16:46.793040 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.793008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tmp-dir\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.793040 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.793044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-dshm\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.793365 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.793080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tls-certs\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.793365 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.793108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-model-cache\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.793365 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.793132    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-home\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.793365 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.793151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6d7\" (UniqueName: \"kubernetes.io/projected/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-kube-api-access-6j6d7\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894373 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894323    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tls-certs\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894588 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-model-cache\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894588 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-home\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894588 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6d7\" (UniqueName: \"kubernetes.io/projected/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-kube-api-access-6j6d7\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894588 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tmp-dir\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894588 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-dshm\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894884 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894788    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-home\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894884 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-model-cache\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.894985 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.894953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tmp-dir\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.896806 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.896784    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-dshm\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.897070 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.897054    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tls-certs\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:46.904389 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:46.904358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6d7\" (UniqueName: \"kubernetes.io/projected/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-kube-api-access-6j6d7\") pod \"auth-test-model-59f108-kserve-bdbd7986c-dsff7\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") " pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:47.014654 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:47.014616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:47.305266 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:47.305225    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"]
Sep 11 16:16:47.307615 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:16:47.307586    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04d2b74f_2cf1_4ae0_9ca1_7839f744fcda.slice/crio-c8bc46fe67f80911cc88ade25ee2829dd5aa46f1dd0a284a6cd9f8eda5ed712c WatchSource:0}: Error finding container c8bc46fe67f80911cc88ade25ee2829dd5aa46f1dd0a284a6cd9f8eda5ed712c: Status 404 returned error can't find the container with id c8bc46fe67f80911cc88ade25ee2829dd5aa46f1dd0a284a6cd9f8eda5ed712c
Sep 11 16:16:47.477451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:47.477411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7" event={"ID":"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda","Type":"ContainerStarted","Data":"5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca"}
Sep 11 16:16:47.477632 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:47.477463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7" event={"ID":"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda","Type":"ContainerStarted","Data":"c8bc46fe67f80911cc88ade25ee2829dd5aa46f1dd0a284a6cd9f8eda5ed712c"}
Sep 11 16:16:47.497736 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:47.497666    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7" podStartSLOduration=1.497644426 podStartE2EDuration="1.497644426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:47.497014405 +0000 UTC m=+1303.920832093" watchObservedRunningTime="2026-09-11 16:16:47.497644426 +0000 UTC m=+1303.921462115"
Sep 11 16:16:48.432790 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:48.432758    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:16:48.433170 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:48.432832    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:16:57.015530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:57.015487    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:57.015530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:57.015542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:57.028556 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:57.028525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:16:57.539114 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:16:57.539083    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:17:17.120180 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.120143    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"]
Sep 11 16:17:17.124005 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.123984    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.129965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.129938    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"auth-test-model-1e0e69-kserve-self-signed-certs\""
Sep 11 16:17:17.129965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.129961    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"openshift-service-ca.crt\""
Sep 11 16:17:17.130195 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.129937    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"kube-root-ca.crt\""
Sep 11 16:17:17.130195 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.129937    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"default-dockercfg-ss8lx\""
Sep 11 16:17:17.141646 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.141613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"]
Sep 11 16:17:17.187004 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.186964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-home\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.187004 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.187009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2eb707-9e84-4b1c-8518-591d9324406d-tls-certs\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.187220 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.187044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-model-cache\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.187220 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.187083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-dshm\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.187220 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.187121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-tmp-dir\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.187220 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.187180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qxr\" (UniqueName: \"kubernetes.io/projected/6a2eb707-9e84-4b1c-8518-591d9324406d-kube-api-access-j2qxr\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288359 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-home\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288564 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2eb707-9e84-4b1c-8518-591d9324406d-tls-certs\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288564 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-model-cache\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288564 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-dshm\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288564 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-tmp-dir\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288791 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qxr\" (UniqueName: \"kubernetes.io/projected/6a2eb707-9e84-4b1c-8518-591d9324406d-kube-api-access-j2qxr\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288849 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-home\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.288908 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.288854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-model-cache\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.289050 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.289027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-tmp-dir\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.290995 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.290969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-dshm\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.291097 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.291063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2eb707-9e84-4b1c-8518-591d9324406d-tls-certs\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.298108 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.298082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qxr\" (UniqueName: \"kubernetes.io/projected/6a2eb707-9e84-4b1c-8518-591d9324406d-kube-api-access-j2qxr\") pod \"auth-test-model-1e0e69-kserve-795b78dd87-7vn2t\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.433783 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.433694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:17.718016 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:17.717987    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"]
Sep 11 16:17:17.720202 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:17:17.720167    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a2eb707_9e84_4b1c_8518_591d9324406d.slice/crio-108382b8f6fcea53a7052be57342cd245fe2d0debaa86891ad2fc6f5700698e9 WatchSource:0}: Error finding container 108382b8f6fcea53a7052be57342cd245fe2d0debaa86891ad2fc6f5700698e9: Status 404 returned error can't find the container with id 108382b8f6fcea53a7052be57342cd245fe2d0debaa86891ad2fc6f5700698e9
Sep 11 16:17:18.636511 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:18.636472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t" event={"ID":"6a2eb707-9e84-4b1c-8518-591d9324406d","Type":"ContainerStarted","Data":"427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867"}
Sep 11 16:17:18.636511 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:18.636519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t" event={"ID":"6a2eb707-9e84-4b1c-8518-591d9324406d","Type":"ContainerStarted","Data":"108382b8f6fcea53a7052be57342cd245fe2d0debaa86891ad2fc6f5700698e9"}
Sep 11 16:17:18.659464 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:18.659400    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t" podStartSLOduration=1.659380429 podStartE2EDuration="1.659380429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:18.65810575 +0000 UTC m=+1335.081923459" watchObservedRunningTime="2026-09-11 16:17:18.659380429 +0000 UTC m=+1335.083198123"
Sep 11 16:17:27.434902 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:27.434821    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:27.434902 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:27.434875    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:27.447615 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:27.447583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:17:27.694342 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:17:27.694261    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:18:00.968890 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:00.968859    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"]
Sep 11 16:18:00.969491 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:00.969065    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5" podUID="57fc771c-d749-4037-9d20-f0c2ba6067bf" containerName="payload-processing" containerID="cri-o://c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe" gracePeriod=30
Sep 11 16:18:01.003589 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.003554    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"]
Sep 11 16:18:01.003780 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.003758    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8" podUID="db4d59a1-acf7-4676-94bc-6574ede54151" containerName="payload-pre-processing" containerID="cri-o://46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711" gracePeriod=30
Sep 11 16:18:01.255129 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.255103    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:18:01.284278 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.284252    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:18:01.331620 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.331588    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drqr6\" (UniqueName: \"kubernetes.io/projected/57fc771c-d749-4037-9d20-f0c2ba6067bf-kube-api-access-drqr6\") pod \"57fc771c-d749-4037-9d20-f0c2ba6067bf\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") "
Sep 11 16:18:01.331810 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.331696    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc771c-d749-4037-9d20-f0c2ba6067bf-plugins-config-volume\") pod \"57fc771c-d749-4037-9d20-f0c2ba6067bf\" (UID: \"57fc771c-d749-4037-9d20-f0c2ba6067bf\") "
Sep 11 16:18:01.331810 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.331738    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d59a1-acf7-4676-94bc-6574ede54151-plugins-config-volume\") pod \"db4d59a1-acf7-4676-94bc-6574ede54151\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") "
Sep 11 16:18:01.331941 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.331855    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvvf7\" (UniqueName: \"kubernetes.io/projected/db4d59a1-acf7-4676-94bc-6574ede54151-kube-api-access-nvvf7\") pod \"db4d59a1-acf7-4676-94bc-6574ede54151\" (UID: \"db4d59a1-acf7-4676-94bc-6574ede54151\") "
Sep 11 16:18:01.332149 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.332117    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fc771c-d749-4037-9d20-f0c2ba6067bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57fc771c-d749-4037-9d20-f0c2ba6067bf" (UID: "57fc771c-d749-4037-9d20-f0c2ba6067bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:01.332296 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.332117    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4d59a1-acf7-4676-94bc-6574ede54151-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db4d59a1-acf7-4676-94bc-6574ede54151" (UID: "db4d59a1-acf7-4676-94bc-6574ede54151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:01.333823 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.333801    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fc771c-d749-4037-9d20-f0c2ba6067bf-kube-api-access-drqr6" (OuterVolumeSpecName: "kube-api-access-drqr6") pod "57fc771c-d749-4037-9d20-f0c2ba6067bf" (UID: "57fc771c-d749-4037-9d20-f0c2ba6067bf"). InnerVolumeSpecName "kube-api-access-drqr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:01.333948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.333922    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4d59a1-acf7-4676-94bc-6574ede54151-kube-api-access-nvvf7" (OuterVolumeSpecName: "kube-api-access-nvvf7") pod "db4d59a1-acf7-4676-94bc-6574ede54151" (UID: "db4d59a1-acf7-4676-94bc-6574ede54151"). InnerVolumeSpecName "kube-api-access-nvvf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:01.432755 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.432718    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d59a1-acf7-4676-94bc-6574ede54151-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.432755 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.432750    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvf7\" (UniqueName: \"kubernetes.io/projected/db4d59a1-acf7-4676-94bc-6574ede54151-kube-api-access-nvvf7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.432755 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.432761    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drqr6\" (UniqueName: \"kubernetes.io/projected/57fc771c-d749-4037-9d20-f0c2ba6067bf-kube-api-access-drqr6\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.432996 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.432769    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc771c-d749-4037-9d20-f0c2ba6067bf-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.857602 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.857565    2559 generic.go:356] "Generic (PLEG): container finished" podID="db4d59a1-acf7-4676-94bc-6574ede54151" containerID="46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711" exitCode=0
Sep 11 16:18:01.857813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.857610    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"
Sep 11 16:18:01.857813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.857654    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8" event={"ID":"db4d59a1-acf7-4676-94bc-6574ede54151","Type":"ContainerDied","Data":"46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711"}
Sep 11 16:18:01.857813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.857710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8" event={"ID":"db4d59a1-acf7-4676-94bc-6574ede54151","Type":"ContainerDied","Data":"d03dc03d5800f8f4ce2f2d20f87001860e9c74db9c7a1b2b28f518e4588df9e5"}
Sep 11 16:18:01.857813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.857738    2559 scope.go:117] "RemoveContainer" containerID="46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711"
Sep 11 16:18:01.859153 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.859128    2559 generic.go:356] "Generic (PLEG): container finished" podID="57fc771c-d749-4037-9d20-f0c2ba6067bf" containerID="c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe" exitCode=0
Sep 11 16:18:01.859296 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.859177    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"
Sep 11 16:18:01.859296 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.859191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5" event={"ID":"57fc771c-d749-4037-9d20-f0c2ba6067bf","Type":"ContainerDied","Data":"c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe"}
Sep 11 16:18:01.859420 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.859323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5" event={"ID":"57fc771c-d749-4037-9d20-f0c2ba6067bf","Type":"ContainerDied","Data":"3a0a8e16c08ad20a9cc68470c977598168d1bad22ae85e3c1c2c1ed6a88d3eda"}
Sep 11 16:18:01.868088 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.868065    2559 scope.go:117] "RemoveContainer" containerID="46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711"
Sep 11 16:18:01.868467 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:18:01.868431    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711\": container with ID starting with 46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711 not found: ID does not exist" containerID="46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711"
Sep 11 16:18:01.868555 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.868473    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711"} err="failed to get container status \"46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711\": rpc error: code = NotFound desc = could not find container \"46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711\": container with ID starting with 46296a4e23499d4105b083d4b3f9b062f0c9af33f7337468087638c4b745b711 not found: ID does not exist"
Sep 11 16:18:01.868555 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.868491    2559 scope.go:117] "RemoveContainer" containerID="c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe"
Sep 11 16:18:01.877230 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.877211    2559 scope.go:117] "RemoveContainer" containerID="c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe"
Sep 11 16:18:01.877518 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:18:01.877498    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe\": container with ID starting with c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe not found: ID does not exist" containerID="c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe"
Sep 11 16:18:01.877587 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.877524    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe"} err="failed to get container status \"c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe\": rpc error: code = NotFound desc = could not find container \"c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe\": container with ID starting with c209184e367ad5f218f382d84a79719df6f8cba8cde915f1b8577f55046903fe not found: ID does not exist"
Sep 11 16:18:01.892112 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.892082    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"]
Sep 11 16:18:01.896650 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.896624    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9da1d032-6d6c6d8c5d-7npw8"]
Sep 11 16:18:01.910706 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.910674    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"]
Sep 11 16:18:01.913251 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:01.913217    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9da1d032-98f8cc94f-46hc5"]
Sep 11 16:18:02.034456 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:02.034412    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57fc771c-d749-4037-9d20-f0c2ba6067bf" path="/var/lib/kubelet/pods/57fc771c-d749-4037-9d20-f0c2ba6067bf/volumes"
Sep 11 16:18:02.034984 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:18:02.034962    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4d59a1-acf7-4676-94bc-6574ede54151" path="/var/lib/kubelet/pods/db4d59a1-acf7-4676-94bc-6574ede54151/volumes"
Sep 11 16:19:36.450496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:36.450416    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"]
Sep 11 16:19:37.601617 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:37.601582    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"]
Sep 11 16:19:51.469387 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.469341    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7" podUID="04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" containerName="main" containerID="cri-o://5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca" gracePeriod=45
Sep 11 16:19:51.716305 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.716277    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:19:51.794310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794280    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-model-cache\") pod \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") "
Sep 11 16:19:51.794481 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794318    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tmp-dir\") pod \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") "
Sep 11 16:19:51.794481 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j6d7\" (UniqueName: \"kubernetes.io/projected/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-kube-api-access-6j6d7\") pod \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") "
Sep 11 16:19:51.794481 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794438    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-home\") pod \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") "
Sep 11 16:19:51.794481 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794463    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-dshm\") pod \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") "
Sep 11 16:19:51.794769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794535    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-model-cache" (OuterVolumeSpecName: "model-cache") pod "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" (UID: "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:51.794769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794552    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" (UID: "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:51.794769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tls-certs\") pod \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\" (UID: \"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda\") "
Sep 11 16:19:51.794769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794688    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-home" (OuterVolumeSpecName: "home") pod "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" (UID: "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:51.794932 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794914    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:51.794979 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794938    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:51.794979 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.794952    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:51.796669 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.796645    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-dshm" (OuterVolumeSpecName: "dshm") pod "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" (UID: "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:51.796736 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.796661    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" (UID: "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:51.796736 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.796678    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-kube-api-access-6j6d7" (OuterVolumeSpecName: "kube-api-access-6j6d7") pod "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" (UID: "04d2b74f-2cf1-4ae0-9ca1-7839f744fcda"). InnerVolumeSpecName "kube-api-access-6j6d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:51.896322 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.896291    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:51.896322 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.896317    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:51.896322 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:51.896327    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j6d7\" (UniqueName: \"kubernetes.io/projected/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda-kube-api-access-6j6d7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:52.426246 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.426211    2559 generic.go:356] "Generic (PLEG): container finished" podID="04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" containerID="5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca" exitCode=0
Sep 11 16:19:52.426460 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.426267    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"
Sep 11 16:19:52.426460 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.426300    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7" event={"ID":"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda","Type":"ContainerDied","Data":"5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca"}
Sep 11 16:19:52.426460 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.426349    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7" event={"ID":"04d2b74f-2cf1-4ae0-9ca1-7839f744fcda","Type":"ContainerDied","Data":"c8bc46fe67f80911cc88ade25ee2829dd5aa46f1dd0a284a6cd9f8eda5ed712c"}
Sep 11 16:19:52.426460 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.426373    2559 scope.go:117] "RemoveContainer" containerID="5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca"
Sep 11 16:19:52.435107 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.435087    2559 scope.go:117] "RemoveContainer" containerID="5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca"
Sep 11 16:19:52.435395 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:19:52.435376    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca\": container with ID starting with 5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca not found: ID does not exist" containerID="5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca"
Sep 11 16:19:52.435471 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.435402    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca"} err="failed to get container status \"5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca\": rpc error: code = NotFound desc = could not find container \"5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca\": container with ID starting with 5b09cfd057c25688bc1613c8716a8ef2eeb79c5865971f834d5f551a6a0082ca not found: ID does not exist"
Sep 11 16:19:52.449231 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.449204    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"]
Sep 11 16:19:52.453769 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.453750    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/auth-test-model-59f108-kserve-bdbd7986c-dsff7"]
Sep 11 16:19:52.621085 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.621043    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t" podUID="6a2eb707-9e84-4b1c-8518-591d9324406d" containerName="main" containerID="cri-o://427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867" gracePeriod=45
Sep 11 16:19:52.876113 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:52.876089    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:19:53.006774 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.006739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-home\") pod \"6a2eb707-9e84-4b1c-8518-591d9324406d\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") "
Sep 11 16:19:53.006932 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.006803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-tmp-dir\") pod \"6a2eb707-9e84-4b1c-8518-591d9324406d\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") "
Sep 11 16:19:53.006932 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.006859    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2eb707-9e84-4b1c-8518-591d9324406d-tls-certs\") pod \"6a2eb707-9e84-4b1c-8518-591d9324406d\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") "
Sep 11 16:19:53.006932 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.006876    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2qxr\" (UniqueName: \"kubernetes.io/projected/6a2eb707-9e84-4b1c-8518-591d9324406d-kube-api-access-j2qxr\") pod \"6a2eb707-9e84-4b1c-8518-591d9324406d\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") "
Sep 11 16:19:53.006932 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.006896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-dshm\") pod \"6a2eb707-9e84-4b1c-8518-591d9324406d\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") "
Sep 11 16:19:53.007097 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.006939    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-model-cache\") pod \"6a2eb707-9e84-4b1c-8518-591d9324406d\" (UID: \"6a2eb707-9e84-4b1c-8518-591d9324406d\") "
Sep 11 16:19:53.007097 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.007043    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-home" (OuterVolumeSpecName: "home") pod "6a2eb707-9e84-4b1c-8518-591d9324406d" (UID: "6a2eb707-9e84-4b1c-8518-591d9324406d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:53.007178 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.007147    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a2eb707-9e84-4b1c-8518-591d9324406d" (UID: "6a2eb707-9e84-4b1c-8518-591d9324406d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:53.007327 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.007304    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a2eb707-9e84-4b1c-8518-591d9324406d" (UID: "6a2eb707-9e84-4b1c-8518-591d9324406d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:53.007410 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.007340    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.007410 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.007369    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.009085 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.009064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2eb707-9e84-4b1c-8518-591d9324406d-kube-api-access-j2qxr" (OuterVolumeSpecName: "kube-api-access-j2qxr") pod "6a2eb707-9e84-4b1c-8518-591d9324406d" (UID: "6a2eb707-9e84-4b1c-8518-591d9324406d"). InnerVolumeSpecName "kube-api-access-j2qxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:53.009191 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.009096    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-dshm" (OuterVolumeSpecName: "dshm") pod "6a2eb707-9e84-4b1c-8518-591d9324406d" (UID: "6a2eb707-9e84-4b1c-8518-591d9324406d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:53.009191 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.009099    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2eb707-9e84-4b1c-8518-591d9324406d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a2eb707-9e84-4b1c-8518-591d9324406d" (UID: "6a2eb707-9e84-4b1c-8518-591d9324406d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:53.107802 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.107766    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.107802 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.107794    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2eb707-9e84-4b1c-8518-591d9324406d-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.108005 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.107803    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qxr\" (UniqueName: \"kubernetes.io/projected/6a2eb707-9e84-4b1c-8518-591d9324406d-kube-api-access-j2qxr\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.108005 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.107824    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2eb707-9e84-4b1c-8518-591d9324406d-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.433520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.433431    2559 generic.go:356] "Generic (PLEG): container finished" podID="6a2eb707-9e84-4b1c-8518-591d9324406d" containerID="427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867" exitCode=0
Sep 11 16:19:53.433520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.433467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t" event={"ID":"6a2eb707-9e84-4b1c-8518-591d9324406d","Type":"ContainerDied","Data":"427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867"}
Sep 11 16:19:53.433520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.433494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t" event={"ID":"6a2eb707-9e84-4b1c-8518-591d9324406d","Type":"ContainerDied","Data":"108382b8f6fcea53a7052be57342cd245fe2d0debaa86891ad2fc6f5700698e9"}
Sep 11 16:19:53.433520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.433468    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"
Sep 11 16:19:53.433520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.433516    2559 scope.go:117] "RemoveContainer" containerID="427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867"
Sep 11 16:19:53.442834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.442817    2559 scope.go:117] "RemoveContainer" containerID="427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867"
Sep 11 16:19:53.443095 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:19:53.443077    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867\": container with ID starting with 427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867 not found: ID does not exist" containerID="427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867"
Sep 11 16:19:53.443172 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.443100    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867"} err="failed to get container status \"427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867\": rpc error: code = NotFound desc = could not find container \"427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867\": container with ID starting with 427adf60e8ee50dd40ec4d53abb7e8d24591f0e46fbddaf364366916e8697867 not found: ID does not exist"
Sep 11 16:19:53.461167 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.461127    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"]
Sep 11 16:19:53.470630 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:53.470596    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/auth-test-model-1e0e69-kserve-795b78dd87-7vn2t"]
Sep 11 16:19:54.034168 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:54.034130    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" path="/var/lib/kubelet/pods/04d2b74f-2cf1-4ae0-9ca1-7839f744fcda/volumes"
Sep 11 16:19:54.034625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:19:54.034608    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a2eb707-9e84-4b1c-8518-591d9324406d" path="/var/lib/kubelet/pods/6a2eb707-9e84-4b1c-8518-591d9324406d/volumes"
Sep 11 16:20:04.911977 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.911939    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"]
Sep 11 16:20:04.912530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912513    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57fc771c-d749-4037-9d20-f0c2ba6067bf" containerName="payload-processing"
Sep 11 16:20:04.912576 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912532    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="57fc771c-d749-4037-9d20-f0c2ba6067bf" containerName="payload-processing"
Sep 11 16:20:04.912576 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912545    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4d59a1-acf7-4676-94bc-6574ede54151" containerName="payload-pre-processing"
Sep 11 16:20:04.912576 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912551    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4d59a1-acf7-4676-94bc-6574ede54151" containerName="payload-pre-processing"
Sep 11 16:20:04.912576 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912562    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" containerName="main"
Sep 11 16:20:04.912576 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912567    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" containerName="main"
Sep 11 16:20:04.912733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912588    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2eb707-9e84-4b1c-8518-591d9324406d" containerName="main"
Sep 11 16:20:04.912733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912593    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2eb707-9e84-4b1c-8518-591d9324406d" containerName="main"
Sep 11 16:20:04.912733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912652    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4d59a1-acf7-4676-94bc-6574ede54151" containerName="payload-pre-processing"
Sep 11 16:20:04.912733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912660    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2eb707-9e84-4b1c-8518-591d9324406d" containerName="main"
Sep 11 16:20:04.912733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912665    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="57fc771c-d749-4037-9d20-f0c2ba6067bf" containerName="payload-processing"
Sep 11 16:20:04.912733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.912672    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d2b74f-2cf1-4ae0-9ca1-7839f744fcda" containerName="main"
Sep 11 16:20:04.914991 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.914973    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:04.919958 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.919935    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"sub-test-model-1e0e69-kserve-self-signed-certs\""
Sep 11 16:20:04.919958 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.919952    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"openshift-service-ca.crt\""
Sep 11 16:20:04.921182 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.921157    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"kube-root-ca.crt\""
Sep 11 16:20:04.921348 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.921208    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"default-dockercfg-ss8lx\""
Sep 11 16:20:04.933451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:04.933425    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"]
Sep 11 16:20:05.013049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.013012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-tmp-dir\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.013049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.013051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-dshm\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.013326 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.013109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c347152d-cfd5-4aad-9ce1-7975ef443230-tls-certs\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.013326 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.013127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-home\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.013326 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.013258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-model-cache\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.013448 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.013366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c48l\" (UniqueName: \"kubernetes.io/projected/c347152d-cfd5-4aad-9ce1-7975ef443230-kube-api-access-5c48l\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.114661 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.114624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c48l\" (UniqueName: \"kubernetes.io/projected/c347152d-cfd5-4aad-9ce1-7975ef443230-kube-api-access-5c48l\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.114853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.114669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-tmp-dir\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.114853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.114697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-dshm\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.114853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.114746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c347152d-cfd5-4aad-9ce1-7975ef443230-tls-certs\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.114853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.114771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-home\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.114853 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.114825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-model-cache\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.115128 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.115020    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-tmp-dir\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.115189 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.115147    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-home\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.115275 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.115228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-model-cache\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.116982 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.116961    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-dshm\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.117207 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.117191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c347152d-cfd5-4aad-9ce1-7975ef443230-tls-certs\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.128671 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.128650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c48l\" (UniqueName: \"kubernetes.io/projected/c347152d-cfd5-4aad-9ce1-7975ef443230-kube-api-access-5c48l\") pod \"sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.224871 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.224787    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:05.500992 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:05.500962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"]
Sep 11 16:20:05.502525 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:20:05.502498    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc347152d_cfd5_4aad_9ce1_7975ef443230.slice/crio-a28cce6c68230193e5fa0cffd2e7fc185cc7f56f4b0a4aaa5b9054061c1c3ae5 WatchSource:0}: Error finding container a28cce6c68230193e5fa0cffd2e7fc185cc7f56f4b0a4aaa5b9054061c1c3ae5: Status 404 returned error can't find the container with id a28cce6c68230193e5fa0cffd2e7fc185cc7f56f4b0a4aaa5b9054061c1c3ae5
Sep 11 16:20:06.501279 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:06.501244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5" event={"ID":"c347152d-cfd5-4aad-9ce1-7975ef443230","Type":"ContainerStarted","Data":"59411493f7b93977efd960129e11faa9d4a99b216c96fb60209eb33fb31b7822"}
Sep 11 16:20:06.501279 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:06.501284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5" event={"ID":"c347152d-cfd5-4aad-9ce1-7975ef443230","Type":"ContainerStarted","Data":"a28cce6c68230193e5fa0cffd2e7fc185cc7f56f4b0a4aaa5b9054061c1c3ae5"}
Sep 11 16:20:06.572674 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:06.572615    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5" podStartSLOduration=2.572599417 podStartE2EDuration="2.572599417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:06.571876486 +0000 UTC m=+1502.995694180" watchObservedRunningTime="2026-09-11 16:20:06.572599417 +0000 UTC m=+1502.996417110"
Sep 11 16:20:07.196907 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.196871    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"]
Sep 11 16:20:07.199473 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.199454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.202778 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.202752    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c98185\""
Sep 11 16:20:07.212083 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.212063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"]
Sep 11 16:20:07.258497 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.258470    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"]
Sep 11 16:20:07.261151 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.261133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.264921 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.264897    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c98185-dockercfg-w2dzh\""
Sep 11 16:20:07.274719 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.274695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"]
Sep 11 16:20:07.336467 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.336432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7df0c8-c587-4563-ac55-cc656d9451e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.336467 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.336467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2q4w\" (UniqueName: \"kubernetes.io/projected/39189a1b-2874-44e1-9c26-2f930d18c829-kube-api-access-s2q4w\") pod \"payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") " pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.336774 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.336559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwv7\" (UniqueName: \"kubernetes.io/projected/7c7df0c8-c587-4563-ac55-cc656d9451e2-kube-api-access-ldwv7\") pod \"payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.336774 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.336661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39189a1b-2874-44e1-9c26-2f930d18c829-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") " pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.437998 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.437962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7df0c8-c587-4563-ac55-cc656d9451e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.437998 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.437998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2q4w\" (UniqueName: \"kubernetes.io/projected/39189a1b-2874-44e1-9c26-2f930d18c829-kube-api-access-s2q4w\") pod \"payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") " pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.438278 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.438034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwv7\" (UniqueName: \"kubernetes.io/projected/7c7df0c8-c587-4563-ac55-cc656d9451e2-kube-api-access-ldwv7\") pod \"payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.438278 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.438058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39189a1b-2874-44e1-9c26-2f930d18c829-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") " pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.438814 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.438790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39189a1b-2874-44e1-9c26-2f930d18c829-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") " pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.438899 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.438794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7df0c8-c587-4563-ac55-cc656d9451e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.450313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.450255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwv7\" (UniqueName: \"kubernetes.io/projected/7c7df0c8-c587-4563-ac55-cc656d9451e2-kube-api-access-ldwv7\") pod \"payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.450435 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.450414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2q4w\" (UniqueName: \"kubernetes.io/projected/39189a1b-2874-44e1-9c26-2f930d18c829-kube-api-access-s2q4w\") pod \"payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") " pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.510343 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.510316    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:07.571633 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.571599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:07.788085 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.788061    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"]
Sep 11 16:20:07.789583 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:20:07.789560    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39189a1b_2874_44e1_9c26_2f930d18c829.slice/crio-fa88dac427a17544f047de848238de425a12aeed153e47bc3abc319075c8bf94 WatchSource:0}: Error finding container fa88dac427a17544f047de848238de425a12aeed153e47bc3abc319075c8bf94: Status 404 returned error can't find the container with id fa88dac427a17544f047de848238de425a12aeed153e47bc3abc319075c8bf94
Sep 11 16:20:07.843359 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:07.843332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"]
Sep 11 16:20:07.844423 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:20:07.844394    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7df0c8_c587_4563_ac55_cc656d9451e2.slice/crio-51f7cbb1693fc223bdbe455149c4c2718f114d5837fee1022bef7c132a04ce59 WatchSource:0}: Error finding container 51f7cbb1693fc223bdbe455149c4c2718f114d5837fee1022bef7c132a04ce59: Status 404 returned error can't find the container with id 51f7cbb1693fc223bdbe455149c4c2718f114d5837fee1022bef7c132a04ce59
Sep 11 16:20:08.514140 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.514096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn" event={"ID":"7c7df0c8-c587-4563-ac55-cc656d9451e2","Type":"ContainerStarted","Data":"325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523"}
Sep 11 16:20:08.514650 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.514163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn" event={"ID":"7c7df0c8-c587-4563-ac55-cc656d9451e2","Type":"ContainerStarted","Data":"51f7cbb1693fc223bdbe455149c4c2718f114d5837fee1022bef7c132a04ce59"}
Sep 11 16:20:08.514650 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.514193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:08.515834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.515805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k" event={"ID":"39189a1b-2874-44e1-9c26-2f930d18c829","Type":"ContainerStarted","Data":"28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac"}
Sep 11 16:20:08.515945 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.515863    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:08.515945 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.515885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k" event={"ID":"39189a1b-2874-44e1-9c26-2f930d18c829","Type":"ContainerStarted","Data":"fa88dac427a17544f047de848238de425a12aeed153e47bc3abc319075c8bf94"}
Sep 11 16:20:08.537496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.537436    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn" podStartSLOduration=1.537420105 podStartE2EDuration="1.537420105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:08.534759447 +0000 UTC m=+1504.958577158" watchObservedRunningTime="2026-09-11 16:20:08.537420105 +0000 UTC m=+1504.961237799"
Sep 11 16:20:08.564361 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:08.564308    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k" podStartSLOduration=1.564291887 podStartE2EDuration="1.564291887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:08.56207759 +0000 UTC m=+1504.985895304" watchObservedRunningTime="2026-09-11 16:20:08.564291887 +0000 UTC m=+1504.988109594"
Sep 11 16:20:15.225597 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:15.225558    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:15.226006 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:15.225610    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:15.238964 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:15.238933    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:15.563523 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:15.563492    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:20:19.523128 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:19.523099    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:20:19.523541 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:19.523171    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:20:40.059531 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.059452    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"]
Sep 11 16:20:40.063116 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.063099    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.074598 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.074576    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bf0614\""
Sep 11 16:20:40.074702 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.074614    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bf0614-dockercfg-jkktk\""
Sep 11 16:20:40.113527 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.113499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"]
Sep 11 16:20:40.135418 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.135387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx25d\" (UniqueName: \"kubernetes.io/projected/d43008af-f9be-4905-a5e7-901c4e705c7e-kube-api-access-kx25d\") pod \"payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.135577 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.135458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43008af-f9be-4905-a5e7-901c4e705c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.235985 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.235949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx25d\" (UniqueName: \"kubernetes.io/projected/d43008af-f9be-4905-a5e7-901c4e705c7e-kube-api-access-kx25d\") pod \"payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.235985 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.235992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43008af-f9be-4905-a5e7-901c4e705c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.236701 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.236681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43008af-f9be-4905-a5e7-901c4e705c7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.257904 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.257871    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx25d\" (UniqueName: \"kubernetes.io/projected/d43008af-f9be-4905-a5e7-901c4e705c7e-kube-api-access-kx25d\") pod \"payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.373676 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.373597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:40.649899 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.649873    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"]
Sep 11 16:20:40.651513 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:20:40.651486    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43008af_f9be_4905_a5e7_901c4e705c7e.slice/crio-cd81d6ed169b9da87cc524891da406607fe0c9e26388daafd8f1cb4c3c85cd67 WatchSource:0}: Error finding container cd81d6ed169b9da87cc524891da406607fe0c9e26388daafd8f1cb4c3c85cd67: Status 404 returned error can't find the container with id cd81d6ed169b9da87cc524891da406607fe0c9e26388daafd8f1cb4c3c85cd67
Sep 11 16:20:40.688812 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:40.688785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876" event={"ID":"d43008af-f9be-4905-a5e7-901c4e705c7e","Type":"ContainerStarted","Data":"cd81d6ed169b9da87cc524891da406607fe0c9e26388daafd8f1cb4c3c85cd67"}
Sep 11 16:20:41.695644 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:41.695593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876" event={"ID":"d43008af-f9be-4905-a5e7-901c4e705c7e","Type":"ContainerStarted","Data":"5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c"}
Sep 11 16:20:41.696068 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:41.695665    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:20:41.726638 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:41.726593    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876" podStartSLOduration=1.7265656649999999 podStartE2EDuration="1.726565665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:41.725434326 +0000 UTC m=+1538.149252037" watchObservedRunningTime="2026-09-11 16:20:41.726565665 +0000 UTC m=+1538.150383359"
Sep 11 16:20:52.704152 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:20:52.704120    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:21:04.228350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:04.228318    2559 scope.go:117] "RemoveContainer" containerID="cc44c8aba8a1c716643a933ddd19f37bee33ed5260d26191a61fceefb5bbd26f"
Sep 11 16:21:11.125704 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.125662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"]
Sep 11 16:21:11.126330 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.125890    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k" podUID="39189a1b-2874-44e1-9c26-2f930d18c829" containerName="payload-processing" containerID="cri-o://28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac" gracePeriod=30
Sep 11 16:21:11.127986 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.127956    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"]
Sep 11 16:21:11.128169 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.128131    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn" podUID="7c7df0c8-c587-4563-ac55-cc656d9451e2" containerName="payload-pre-processing" containerID="cri-o://325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523" gracePeriod=30
Sep 11 16:21:11.392017 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.391961    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:21:11.412640 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.412619    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:21:11.443993 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.443954    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7df0c8-c587-4563-ac55-cc656d9451e2-plugins-config-volume\") pod \"7c7df0c8-c587-4563-ac55-cc656d9451e2\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") "
Sep 11 16:21:11.444154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.444111    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39189a1b-2874-44e1-9c26-2f930d18c829-plugins-config-volume\") pod \"39189a1b-2874-44e1-9c26-2f930d18c829\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") "
Sep 11 16:21:11.444269 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.444156    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldwv7\" (UniqueName: \"kubernetes.io/projected/7c7df0c8-c587-4563-ac55-cc656d9451e2-kube-api-access-ldwv7\") pod \"7c7df0c8-c587-4563-ac55-cc656d9451e2\" (UID: \"7c7df0c8-c587-4563-ac55-cc656d9451e2\") "
Sep 11 16:21:11.444494 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.444466    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7df0c8-c587-4563-ac55-cc656d9451e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c7df0c8-c587-4563-ac55-cc656d9451e2" (UID: "7c7df0c8-c587-4563-ac55-cc656d9451e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:11.444589 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.444558    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39189a1b-2874-44e1-9c26-2f930d18c829-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39189a1b-2874-44e1-9c26-2f930d18c829" (UID: "39189a1b-2874-44e1-9c26-2f930d18c829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:11.446355 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.446330    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7df0c8-c587-4563-ac55-cc656d9451e2-kube-api-access-ldwv7" (OuterVolumeSpecName: "kube-api-access-ldwv7") pod "7c7df0c8-c587-4563-ac55-cc656d9451e2" (UID: "7c7df0c8-c587-4563-ac55-cc656d9451e2"). InnerVolumeSpecName "kube-api-access-ldwv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:11.544845 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.544805    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2q4w\" (UniqueName: \"kubernetes.io/projected/39189a1b-2874-44e1-9c26-2f930d18c829-kube-api-access-s2q4w\") pod \"39189a1b-2874-44e1-9c26-2f930d18c829\" (UID: \"39189a1b-2874-44e1-9c26-2f930d18c829\") "
Sep 11 16:21:11.545049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.545004    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7df0c8-c587-4563-ac55-cc656d9451e2-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.545049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.545017    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39189a1b-2874-44e1-9c26-2f930d18c829-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.545049 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.545026    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwv7\" (UniqueName: \"kubernetes.io/projected/7c7df0c8-c587-4563-ac55-cc656d9451e2-kube-api-access-ldwv7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.546983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.546951    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39189a1b-2874-44e1-9c26-2f930d18c829-kube-api-access-s2q4w" (OuterVolumeSpecName: "kube-api-access-s2q4w") pod "39189a1b-2874-44e1-9c26-2f930d18c829" (UID: "39189a1b-2874-44e1-9c26-2f930d18c829"). InnerVolumeSpecName "kube-api-access-s2q4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:11.645989 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.645903    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2q4w\" (UniqueName: \"kubernetes.io/projected/39189a1b-2874-44e1-9c26-2f930d18c829-kube-api-access-s2q4w\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.855524 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.855492    2559 generic.go:356] "Generic (PLEG): container finished" podID="7c7df0c8-c587-4563-ac55-cc656d9451e2" containerID="325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523" exitCode=0
Sep 11 16:21:11.855706 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.855535    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"
Sep 11 16:21:11.855706 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.855585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn" event={"ID":"7c7df0c8-c587-4563-ac55-cc656d9451e2","Type":"ContainerDied","Data":"325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523"}
Sep 11 16:21:11.855706 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.855632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn" event={"ID":"7c7df0c8-c587-4563-ac55-cc656d9451e2","Type":"ContainerDied","Data":"51f7cbb1693fc223bdbe455149c4c2718f114d5837fee1022bef7c132a04ce59"}
Sep 11 16:21:11.855706 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.855658    2559 scope.go:117] "RemoveContainer" containerID="325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523"
Sep 11 16:21:11.857046 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.857020    2559 generic.go:356] "Generic (PLEG): container finished" podID="39189a1b-2874-44e1-9c26-2f930d18c829" containerID="28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac" exitCode=0
Sep 11 16:21:11.857149 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.857068    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"
Sep 11 16:21:11.857149 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.857090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k" event={"ID":"39189a1b-2874-44e1-9c26-2f930d18c829","Type":"ContainerDied","Data":"28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac"}
Sep 11 16:21:11.857149 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.857126    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k" event={"ID":"39189a1b-2874-44e1-9c26-2f930d18c829","Type":"ContainerDied","Data":"fa88dac427a17544f047de848238de425a12aeed153e47bc3abc319075c8bf94"}
Sep 11 16:21:11.864455 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.864436    2559 scope.go:117] "RemoveContainer" containerID="325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523"
Sep 11 16:21:11.864749 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:21:11.864730    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523\": container with ID starting with 325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523 not found: ID does not exist" containerID="325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523"
Sep 11 16:21:11.864819 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.864755    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523"} err="failed to get container status \"325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523\": rpc error: code = NotFound desc = could not find container \"325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523\": container with ID starting with 325cef5d5b69e3a265d03ce2955db633ee126bd7d0e2a1689dc2dfc542dc9523 not found: ID does not exist"
Sep 11 16:21:11.864819 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.864772    2559 scope.go:117] "RemoveContainer" containerID="28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac"
Sep 11 16:21:11.873065 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.873048    2559 scope.go:117] "RemoveContainer" containerID="28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac"
Sep 11 16:21:11.873508 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:21:11.873491    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac\": container with ID starting with 28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac not found: ID does not exist" containerID="28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac"
Sep 11 16:21:11.873564 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.873513    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac"} err="failed to get container status \"28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac\": rpc error: code = NotFound desc = could not find container \"28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac\": container with ID starting with 28a163e76c37a79187d0f862d73592199f09f4ac569beac3401b0fa0781e44ac not found: ID does not exist"
Sep 11 16:21:11.889639 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.889616    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"]
Sep 11 16:21:11.897175 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.897127    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c98185-b79466fc4-p57wn"]
Sep 11 16:21:11.918923 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.918901    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"]
Sep 11 16:21:11.923246 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:11.923218    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c98185-548f4cf9d7-cmn9k"]
Sep 11 16:21:12.034766 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:12.034737    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39189a1b-2874-44e1-9c26-2f930d18c829" path="/var/lib/kubelet/pods/39189a1b-2874-44e1-9c26-2f930d18c829/volumes"
Sep 11 16:21:12.035119 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:12.035106    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c7df0c8-c587-4563-ac55-cc656d9451e2" path="/var/lib/kubelet/pods/7c7df0c8-c587-4563-ac55-cc656d9451e2/volumes"
Sep 11 16:21:15.722541 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:15.722501    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"]
Sep 11 16:21:19.483304 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.483268    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"]
Sep 11 16:21:19.483717 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.483704    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39189a1b-2874-44e1-9c26-2f930d18c829" containerName="payload-processing"
Sep 11 16:21:19.483761 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.483718    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39189a1b-2874-44e1-9c26-2f930d18c829" containerName="payload-processing"
Sep 11 16:21:19.483761 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.483729    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7df0c8-c587-4563-ac55-cc656d9451e2" containerName="payload-pre-processing"
Sep 11 16:21:19.483761 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.483734    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7df0c8-c587-4563-ac55-cc656d9451e2" containerName="payload-pre-processing"
Sep 11 16:21:19.483855 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.483816    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39189a1b-2874-44e1-9c26-2f930d18c829" containerName="payload-processing"
Sep 11 16:21:19.483855 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.483827    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7df0c8-c587-4563-ac55-cc656d9451e2" containerName="payload-pre-processing"
Sep 11 16:21:19.488539 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.488521    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"
Sep 11 16:21:19.518903 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.518875    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:21:19.543676 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.543649    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"]
Sep 11 16:21:19.621401 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.621369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc59ba0f-3e10-418b-b342-94878126cc7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m\" (UID: \"dc59ba0f-3e10-418b-b342-94878126cc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"
Sep 11 16:21:19.722518 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.722478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc59ba0f-3e10-418b-b342-94878126cc7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m\" (UID: \"dc59ba0f-3e10-418b-b342-94878126cc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"
Sep 11 16:21:19.723119 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.723098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc59ba0f-3e10-418b-b342-94878126cc7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m\" (UID: \"dc59ba0f-3e10-418b-b342-94878126cc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"
Sep 11 16:21:19.798301 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:19.798263    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"
Sep 11 16:21:20.074112 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:20.074089    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"]
Sep 11 16:21:20.076071 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:21:20.076047    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc59ba0f_3e10_418b_b342_94878126cc7d.slice/crio-6d7bb9068267b8e90565b9a1589a96a1d29e0fe48b89da51da28981624925a98 WatchSource:0}: Error finding container 6d7bb9068267b8e90565b9a1589a96a1d29e0fe48b89da51da28981624925a98: Status 404 returned error can't find the container with id 6d7bb9068267b8e90565b9a1589a96a1d29e0fe48b89da51da28981624925a98
Sep 11 16:21:20.912439 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:20.912404    2559 generic.go:356] "Generic (PLEG): container finished" podID="dc59ba0f-3e10-418b-b342-94878126cc7d" containerID="c508251050cc223bf61440aaeb77b5408350a89c20abc346d584caf4d798a0bc" exitCode=0
Sep 11 16:21:20.912813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:20.912493    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m" event={"ID":"dc59ba0f-3e10-418b-b342-94878126cc7d","Type":"ContainerDied","Data":"c508251050cc223bf61440aaeb77b5408350a89c20abc346d584caf4d798a0bc"}
Sep 11 16:21:20.912813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:20.912543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m" event={"ID":"dc59ba0f-3e10-418b-b342-94878126cc7d","Type":"ContainerStarted","Data":"6d7bb9068267b8e90565b9a1589a96a1d29e0fe48b89da51da28981624925a98"}
Sep 11 16:21:22.049646 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:22.049621    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"
Sep 11 16:21:22.143921 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:22.143887    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc59ba0f-3e10-418b-b342-94878126cc7d-maas-api-service-ca\") pod \"dc59ba0f-3e10-418b-b342-94878126cc7d\" (UID: \"dc59ba0f-3e10-418b-b342-94878126cc7d\") "
Sep 11 16:21:22.144227 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:22.144199    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc59ba0f-3e10-418b-b342-94878126cc7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc59ba0f-3e10-418b-b342-94878126cc7d" (UID: "dc59ba0f-3e10-418b-b342-94878126cc7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:22.245042 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:22.245014    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc59ba0f-3e10-418b-b342-94878126cc7d-maas-api-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:22.923208 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:22.923170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m" event={"ID":"dc59ba0f-3e10-418b-b342-94878126cc7d","Type":"ContainerDied","Data":"6d7bb9068267b8e90565b9a1589a96a1d29e0fe48b89da51da28981624925a98"}
Sep 11 16:21:22.923425 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:22.923225    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d7bb9068267b8e90565b9a1589a96a1d29e0fe48b89da51da28981624925a98"
Sep 11 16:21:22.923425 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:22.923188    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"
Sep 11 16:21:26.017823 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.017790    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"]
Sep 11 16:21:26.018260 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.018223    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc59ba0f-3e10-418b-b342-94878126cc7d" containerName="revoke-keys"
Sep 11 16:21:26.018260 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.018251    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc59ba0f-3e10-418b-b342-94878126cc7d" containerName="revoke-keys"
Sep 11 16:21:26.018352 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.018319    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc59ba0f-3e10-418b-b342-94878126cc7d" containerName="revoke-keys"
Sep 11 16:21:26.021401 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.021384    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.025995 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.025966    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"kube-root-ca.crt\""
Sep 11 16:21:26.025995 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.025989    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"openshift-service-ca.crt\""
Sep 11 16:21:26.026181 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.025966    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"rate-test-model-59f108-kserve-self-signed-certs\""
Sep 11 16:21:26.026181 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.025996    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"default-dockercfg-9kwxq\""
Sep 11 16:21:26.035797 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.035772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"]
Sep 11 16:21:26.180879 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.180841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-dshm\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.181053 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.180933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-home\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.181053 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.180964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-tmp-dir\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.181053 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.180981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4a464eb-3152-4d67-bc99-b22de0089e2d-tls-certs\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.181053 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.181004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42wf\" (UniqueName: \"kubernetes.io/projected/a4a464eb-3152-4d67-bc99-b22de0089e2d-kube-api-access-s42wf\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.181198 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.181125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-model-cache\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-dshm\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-home\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282334 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-tmp-dir\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282623 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282347    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4a464eb-3152-4d67-bc99-b22de0089e2d-tls-certs\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282623 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42wf\" (UniqueName: \"kubernetes.io/projected/a4a464eb-3152-4d67-bc99-b22de0089e2d-kube-api-access-s42wf\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282623 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-model-cache\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282776 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-home\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282880 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-tmp-dir\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.282880 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.282872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-model-cache\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.284548 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.284530    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-dshm\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.284835 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.284807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4a464eb-3152-4d67-bc99-b22de0089e2d-tls-certs\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.291493 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.291472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42wf\" (UniqueName: \"kubernetes.io/projected/a4a464eb-3152-4d67-bc99-b22de0089e2d-kube-api-access-s42wf\") pod \"rate-test-model-59f108-kserve-6cd65b999f-prg4w\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") " pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.331070 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.331046    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:26.604708 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.604682    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"]
Sep 11 16:21:26.606224 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:21:26.606193    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4a464eb_3152_4d67_bc99_b22de0089e2d.slice/crio-dbaaa21f18e5ee4068372051098548c6f98a8f469bd995811f1697f7bb93f466 WatchSource:0}: Error finding container dbaaa21f18e5ee4068372051098548c6f98a8f469bd995811f1697f7bb93f466: Status 404 returned error can't find the container with id dbaaa21f18e5ee4068372051098548c6f98a8f469bd995811f1697f7bb93f466
Sep 11 16:21:26.945261 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.945140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w" event={"ID":"a4a464eb-3152-4d67-bc99-b22de0089e2d","Type":"ContainerStarted","Data":"67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e"}
Sep 11 16:21:26.945261 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.945185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w" event={"ID":"a4a464eb-3152-4d67-bc99-b22de0089e2d","Type":"ContainerStarted","Data":"dbaaa21f18e5ee4068372051098548c6f98a8f469bd995811f1697f7bb93f466"}
Sep 11 16:21:26.975434 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:26.975377    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w" podStartSLOduration=1.975361651 podStartE2EDuration="1.975361651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:26.973697175 +0000 UTC m=+1583.397514870" watchObservedRunningTime="2026-09-11 16:21:26.975361651 +0000 UTC m=+1583.399179372"
Sep 11 16:21:30.741013 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:30.740946    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5" podUID="c347152d-cfd5-4aad-9ce1-7975ef443230" containerName="main" containerID="cri-o://59411493f7b93977efd960129e11faa9d4a99b216c96fb60209eb33fb31b7822" gracePeriod=45
Sep 11 16:21:30.970603 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:30.970573    2559 generic.go:356] "Generic (PLEG): container finished" podID="c347152d-cfd5-4aad-9ce1-7975ef443230" containerID="59411493f7b93977efd960129e11faa9d4a99b216c96fb60209eb33fb31b7822" exitCode=0
Sep 11 16:21:30.970690 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:30.970645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5" event={"ID":"c347152d-cfd5-4aad-9ce1-7975ef443230","Type":"ContainerDied","Data":"59411493f7b93977efd960129e11faa9d4a99b216c96fb60209eb33fb31b7822"}
Sep 11 16:21:30.991697 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:30.991643    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:21:31.131976 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.131933    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-tmp-dir\") pod \"c347152d-cfd5-4aad-9ce1-7975ef443230\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") "
Sep 11 16:21:31.132178 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132000    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-dshm\") pod \"c347152d-cfd5-4aad-9ce1-7975ef443230\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") "
Sep 11 16:21:31.132178 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132082    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-home\") pod \"c347152d-cfd5-4aad-9ce1-7975ef443230\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") "
Sep 11 16:21:31.132178 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132112    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c347152d-cfd5-4aad-9ce1-7975ef443230-tls-certs\") pod \"c347152d-cfd5-4aad-9ce1-7975ef443230\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") "
Sep 11 16:21:31.132178 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132145    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c48l\" (UniqueName: \"kubernetes.io/projected/c347152d-cfd5-4aad-9ce1-7975ef443230-kube-api-access-5c48l\") pod \"c347152d-cfd5-4aad-9ce1-7975ef443230\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") "
Sep 11 16:21:31.132432 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132183    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-model-cache\") pod \"c347152d-cfd5-4aad-9ce1-7975ef443230\" (UID: \"c347152d-cfd5-4aad-9ce1-7975ef443230\") "
Sep 11 16:21:31.132432 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132279    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c347152d-cfd5-4aad-9ce1-7975ef443230" (UID: "c347152d-cfd5-4aad-9ce1-7975ef443230"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:31.132432 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132401    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-home" (OuterVolumeSpecName: "home") pod "c347152d-cfd5-4aad-9ce1-7975ef443230" (UID: "c347152d-cfd5-4aad-9ce1-7975ef443230"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:31.132544 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132524    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-model-cache" (OuterVolumeSpecName: "model-cache") pod "c347152d-cfd5-4aad-9ce1-7975ef443230" (UID: "c347152d-cfd5-4aad-9ce1-7975ef443230"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:31.132622 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132603    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:31.132676 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132624    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:31.132676 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.132635    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:31.134173 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.134145    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-dshm" (OuterVolumeSpecName: "dshm") pod "c347152d-cfd5-4aad-9ce1-7975ef443230" (UID: "c347152d-cfd5-4aad-9ce1-7975ef443230"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:31.134288 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.134250    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c347152d-cfd5-4aad-9ce1-7975ef443230-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c347152d-cfd5-4aad-9ce1-7975ef443230" (UID: "c347152d-cfd5-4aad-9ce1-7975ef443230"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:31.134371 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.134350    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c347152d-cfd5-4aad-9ce1-7975ef443230-kube-api-access-5c48l" (OuterVolumeSpecName: "kube-api-access-5c48l") pod "c347152d-cfd5-4aad-9ce1-7975ef443230" (UID: "c347152d-cfd5-4aad-9ce1-7975ef443230"). InnerVolumeSpecName "kube-api-access-5c48l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:31.233732 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.233699    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c347152d-cfd5-4aad-9ce1-7975ef443230-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:31.233732 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.233729    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c48l\" (UniqueName: \"kubernetes.io/projected/c347152d-cfd5-4aad-9ce1-7975ef443230-kube-api-access-5c48l\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:31.233732 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.233739    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c347152d-cfd5-4aad-9ce1-7975ef443230-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:31.977570 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.977541    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"
Sep 11 16:21:31.977942 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.977540    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5" event={"ID":"c347152d-cfd5-4aad-9ce1-7975ef443230","Type":"ContainerDied","Data":"a28cce6c68230193e5fa0cffd2e7fc185cc7f56f4b0a4aaa5b9054061c1c3ae5"}
Sep 11 16:21:31.977942 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:31.977670    2559 scope.go:117] "RemoveContainer" containerID="59411493f7b93977efd960129e11faa9d4a99b216c96fb60209eb33fb31b7822"
Sep 11 16:21:32.004325 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:32.004303    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"]
Sep 11 16:21:32.013173 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:32.013146    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/sub-test-model-1e0e69-kserve-5bcf5464c-7ffk5"]
Sep 11 16:21:32.034598 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:32.034569    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c347152d-cfd5-4aad-9ce1-7975ef443230" path="/var/lib/kubelet/pods/c347152d-cfd5-4aad-9ce1-7975ef443230/volumes"
Sep 11 16:21:35.183901 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.183860    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"]
Sep 11 16:21:35.184310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.184122    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876" podUID="d43008af-f9be-4905-a5e7-901c4e705c7e" containerName="payload-pre-processing" containerID="cri-o://5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c" gracePeriod=30
Sep 11 16:21:35.442544 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.442486    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:21:35.501558 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.501520    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"]
Sep 11 16:21:35.508530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.508500    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf0614-fmt4m"]
Sep 11 16:21:35.576376 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.576346    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx25d\" (UniqueName: \"kubernetes.io/projected/d43008af-f9be-4905-a5e7-901c4e705c7e-kube-api-access-kx25d\") pod \"d43008af-f9be-4905-a5e7-901c4e705c7e\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") "
Sep 11 16:21:35.576503 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.576388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43008af-f9be-4905-a5e7-901c4e705c7e-plugins-config-volume\") pod \"d43008af-f9be-4905-a5e7-901c4e705c7e\" (UID: \"d43008af-f9be-4905-a5e7-901c4e705c7e\") "
Sep 11 16:21:35.576784 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.576761    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43008af-f9be-4905-a5e7-901c4e705c7e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d43008af-f9be-4905-a5e7-901c4e705c7e" (UID: "d43008af-f9be-4905-a5e7-901c4e705c7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:35.578394 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.578375    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43008af-f9be-4905-a5e7-901c4e705c7e-kube-api-access-kx25d" (OuterVolumeSpecName: "kube-api-access-kx25d") pod "d43008af-f9be-4905-a5e7-901c4e705c7e" (UID: "d43008af-f9be-4905-a5e7-901c4e705c7e"). InnerVolumeSpecName "kube-api-access-kx25d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:35.677629 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.677600    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx25d\" (UniqueName: \"kubernetes.io/projected/d43008af-f9be-4905-a5e7-901c4e705c7e-kube-api-access-kx25d\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.677629 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.677626    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43008af-f9be-4905-a5e7-901c4e705c7e-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.999781 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.999750    2559 generic.go:356] "Generic (PLEG): container finished" podID="d43008af-f9be-4905-a5e7-901c4e705c7e" containerID="5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c" exitCode=0
Sep 11 16:21:35.999995 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.999793    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"
Sep 11 16:21:35.999995 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.999835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876" event={"ID":"d43008af-f9be-4905-a5e7-901c4e705c7e","Type":"ContainerDied","Data":"5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c"}
Sep 11 16:21:35.999995 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.999892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876" event={"ID":"d43008af-f9be-4905-a5e7-901c4e705c7e","Type":"ContainerDied","Data":"cd81d6ed169b9da87cc524891da406607fe0c9e26388daafd8f1cb4c3c85cd67"}
Sep 11 16:21:35.999995 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:35.999923    2559 scope.go:117] "RemoveContainer" containerID="5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c"
Sep 11 16:21:36.008866 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.008851    2559 scope.go:117] "RemoveContainer" containerID="5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c"
Sep 11 16:21:36.009126 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:21:36.009109    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c\": container with ID starting with 5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c not found: ID does not exist" containerID="5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c"
Sep 11 16:21:36.009174 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.009134    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c"} err="failed to get container status \"5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c\": rpc error: code = NotFound desc = could not find container \"5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c\": container with ID starting with 5e05835037674b511430c2385c751d45cb985a32b5d3db6eb8615815c85c542c not found: ID does not exist"
Sep 11 16:21:36.036384 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.036359    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc59ba0f-3e10-418b-b342-94878126cc7d" path="/var/lib/kubelet/pods/dc59ba0f-3e10-418b-b342-94878126cc7d/volumes"
Sep 11 16:21:36.036638 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.036624    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"]
Sep 11 16:21:36.036681 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.036663    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf0614-6db5776b46-46876"]
Sep 11 16:21:36.332301 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.332192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:36.332301 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.332270    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:36.344736 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:36.344711    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:37.017819 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:37.017793    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:21:38.034008 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:38.033971    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d43008af-f9be-4905-a5e7-901c4e705c7e" path="/var/lib/kubelet/pods/d43008af-f9be-4905-a5e7-901c4e705c7e/volumes"
Sep 11 16:21:42.884524 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.884484    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"]
Sep 11 16:21:42.885018 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.884893    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43008af-f9be-4905-a5e7-901c4e705c7e" containerName="payload-pre-processing"
Sep 11 16:21:42.885018 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.884904    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43008af-f9be-4905-a5e7-901c4e705c7e" containerName="payload-pre-processing"
Sep 11 16:21:42.885018 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.884918    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c347152d-cfd5-4aad-9ce1-7975ef443230" containerName="main"
Sep 11 16:21:42.885018 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.884924    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c347152d-cfd5-4aad-9ce1-7975ef443230" containerName="main"
Sep 11 16:21:42.885018 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.884992    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c347152d-cfd5-4aad-9ce1-7975ef443230" containerName="main"
Sep 11 16:21:42.885018 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.885004    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43008af-f9be-4905-a5e7-901c4e705c7e" containerName="payload-pre-processing"
Sep 11 16:21:42.889171 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.889152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"
Sep 11 16:21:42.892086 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.892066    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:21:42.897282 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.897258    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"]
Sep 11 16:21:42.935799 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:42.935756    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18e40b-ad28-4a3c-9235-9a6dae240b8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r\" (UID: \"ad18e40b-ad28-4a3c-9235-9a6dae240b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"
Sep 11 16:21:43.036381 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:43.036345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18e40b-ad28-4a3c-9235-9a6dae240b8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r\" (UID: \"ad18e40b-ad28-4a3c-9235-9a6dae240b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"
Sep 11 16:21:43.036948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:43.036929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18e40b-ad28-4a3c-9235-9a6dae240b8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r\" (UID: \"ad18e40b-ad28-4a3c-9235-9a6dae240b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"
Sep 11 16:21:43.200776 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:43.200698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"
Sep 11 16:21:43.473068 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:43.473038    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"]
Sep 11 16:21:43.474076 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:21:43.474051    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad18e40b_ad28_4a3c_9235_9a6dae240b8c.slice/crio-ac0e8bd46619e8a29a978f462013d3b190865cabb0951b007b03002903a6f8f6 WatchSource:0}: Error finding container ac0e8bd46619e8a29a978f462013d3b190865cabb0951b007b03002903a6f8f6: Status 404 returned error can't find the container with id ac0e8bd46619e8a29a978f462013d3b190865cabb0951b007b03002903a6f8f6
Sep 11 16:21:44.041503 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:44.041467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r" event={"ID":"ad18e40b-ad28-4a3c-9235-9a6dae240b8c","Type":"ContainerStarted","Data":"a69fe3f8038e32d9fca4f48c917e0dcbf2e5c5b2ccf9da953eb000f4d8352597"}
Sep 11 16:21:44.041879 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:44.041528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r" event={"ID":"ad18e40b-ad28-4a3c-9235-9a6dae240b8c","Type":"ContainerStarted","Data":"ac0e8bd46619e8a29a978f462013d3b190865cabb0951b007b03002903a6f8f6"}
Sep 11 16:21:45.048402 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:45.048362    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad18e40b-ad28-4a3c-9235-9a6dae240b8c" containerID="a69fe3f8038e32d9fca4f48c917e0dcbf2e5c5b2ccf9da953eb000f4d8352597" exitCode=0
Sep 11 16:21:45.048777 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:45.048408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r" event={"ID":"ad18e40b-ad28-4a3c-9235-9a6dae240b8c","Type":"ContainerDied","Data":"a69fe3f8038e32d9fca4f48c917e0dcbf2e5c5b2ccf9da953eb000f4d8352597"}
Sep 11 16:21:46.187036 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:46.187008    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"
Sep 11 16:21:46.264546 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:46.264513    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18e40b-ad28-4a3c-9235-9a6dae240b8c-maas-api-service-ca\") pod \"ad18e40b-ad28-4a3c-9235-9a6dae240b8c\" (UID: \"ad18e40b-ad28-4a3c-9235-9a6dae240b8c\") "
Sep 11 16:21:46.264867 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:46.264844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad18e40b-ad28-4a3c-9235-9a6dae240b8c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad18e40b-ad28-4a3c-9235-9a6dae240b8c" (UID: "ad18e40b-ad28-4a3c-9235-9a6dae240b8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:46.365390 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:46.365314    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18e40b-ad28-4a3c-9235-9a6dae240b8c-maas-api-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:47.060311 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:47.060275    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"
Sep 11 16:21:47.060311 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:47.060299    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r" event={"ID":"ad18e40b-ad28-4a3c-9235-9a6dae240b8c","Type":"ContainerDied","Data":"ac0e8bd46619e8a29a978f462013d3b190865cabb0951b007b03002903a6f8f6"}
Sep 11 16:21:47.060526 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:47.060346    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac0e8bd46619e8a29a978f462013d3b190865cabb0951b007b03002903a6f8f6"
Sep 11 16:21:53.818741 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:53.818710    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"]
Sep 11 16:21:53.819174 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:53.818937    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4" podUID="2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" containerName="payload-processing" containerID="cri-o://2c7f9bcbda46bcf0f131b30bdb34ad6403b1705a2601e886c6540db3c0dfe10e" gracePeriod=30
Sep 11 16:21:53.827833 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:53.827805    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"]
Sep 11 16:21:53.828193 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:53.828161    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8" podUID="a143c913-c93b-459d-bf1b-1cce98e8323f" containerName="payload-pre-processing" containerID="cri-o://80e3fde2b47262af6df81aaaf4eb590b3a9af11839d0bfcf32ccef93288d2ff8" gracePeriod=30
Sep 11 16:21:54.096442 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.096412    2559 generic.go:356] "Generic (PLEG): container finished" podID="a143c913-c93b-459d-bf1b-1cce98e8323f" containerID="80e3fde2b47262af6df81aaaf4eb590b3a9af11839d0bfcf32ccef93288d2ff8" exitCode=0
Sep 11 16:21:54.096542 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.096482    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8" event={"ID":"a143c913-c93b-459d-bf1b-1cce98e8323f","Type":"ContainerDied","Data":"80e3fde2b47262af6df81aaaf4eb590b3a9af11839d0bfcf32ccef93288d2ff8"}
Sep 11 16:21:54.096542 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.096535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8" event={"ID":"a143c913-c93b-459d-bf1b-1cce98e8323f","Type":"ContainerDied","Data":"26f2e588b0ae76b760910303bd9839685b3ceebc0d41190b07f9db41c4c1fb91"}
Sep 11 16:21:54.096643 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.096552    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f2e588b0ae76b760910303bd9839685b3ceebc0d41190b07f9db41c4c1fb91"
Sep 11 16:21:54.097762 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.097743    2559 generic.go:356] "Generic (PLEG): container finished" podID="2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" containerID="2c7f9bcbda46bcf0f131b30bdb34ad6403b1705a2601e886c6540db3c0dfe10e" exitCode=0
Sep 11 16:21:54.097855 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.097817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4" event={"ID":"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf","Type":"ContainerDied","Data":"2c7f9bcbda46bcf0f131b30bdb34ad6403b1705a2601e886c6540db3c0dfe10e"}
Sep 11 16:21:54.097894 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.097856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4" event={"ID":"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf","Type":"ContainerDied","Data":"9548b727539f8478d6983c97d64afaa26109b764b58d159ef982ab944caad063"}
Sep 11 16:21:54.097894 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.097889    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9548b727539f8478d6983c97d64afaa26109b764b58d159ef982ab944caad063"
Sep 11 16:21:54.102738 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.102718    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:21:54.105979 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.105963    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:21:54.120607 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.120580    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"]
Sep 11 16:21:54.137482 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.137457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a143c913-c93b-459d-bf1b-1cce98e8323f-plugins-config-volume\") pod \"a143c913-c93b-459d-bf1b-1cce98e8323f\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") "
Sep 11 16:21:54.137635 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.137519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-plugins-config-volume\") pod \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") "
Sep 11 16:21:54.137635 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.137551    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmqzp\" (UniqueName: \"kubernetes.io/projected/a143c913-c93b-459d-bf1b-1cce98e8323f-kube-api-access-gmqzp\") pod \"a143c913-c93b-459d-bf1b-1cce98e8323f\" (UID: \"a143c913-c93b-459d-bf1b-1cce98e8323f\") "
Sep 11 16:21:54.137635 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.137583    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cql6\" (UniqueName: \"kubernetes.io/projected/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-kube-api-access-6cql6\") pod \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\" (UID: \"2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf\") "
Sep 11 16:21:54.137994 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.137952    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a143c913-c93b-459d-bf1b-1cce98e8323f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a143c913-c93b-459d-bf1b-1cce98e8323f" (UID: "a143c913-c93b-459d-bf1b-1cce98e8323f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:54.138095 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.138018    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" (UID: "2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:54.140601 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.140578    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a143c913-c93b-459d-bf1b-1cce98e8323f-kube-api-access-gmqzp" (OuterVolumeSpecName: "kube-api-access-gmqzp") pod "a143c913-c93b-459d-bf1b-1cce98e8323f" (UID: "a143c913-c93b-459d-bf1b-1cce98e8323f"). InnerVolumeSpecName "kube-api-access-gmqzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:54.140783 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.140756    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-kube-api-access-6cql6" (OuterVolumeSpecName: "kube-api-access-6cql6") pod "2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" (UID: "2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf"). InnerVolumeSpecName "kube-api-access-6cql6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:54.142580 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.142556    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f7c21d-klq7r"]
Sep 11 16:21:54.239339 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.239308    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a143c913-c93b-459d-bf1b-1cce98e8323f-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:54.239339 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.239333    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:54.239339 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.239344    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmqzp\" (UniqueName: \"kubernetes.io/projected/a143c913-c93b-459d-bf1b-1cce98e8323f-kube-api-access-gmqzp\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:54.239567 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:54.239352    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cql6\" (UniqueName: \"kubernetes.io/projected/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf-kube-api-access-6cql6\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:21:55.103116 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:55.103034    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"
Sep 11 16:21:55.103501 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:55.103034    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"
Sep 11 16:21:55.132379 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:55.132351    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"]
Sep 11 16:21:55.136259 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:55.136217    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f7c21d-65ccc75dcf-vc4z4"]
Sep 11 16:21:55.151507 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:55.151483    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"]
Sep 11 16:21:55.154202 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:55.154170    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f7c21d-5cdd6f7cb6-cb7h8"]
Sep 11 16:21:56.034139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:56.034105    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" path="/var/lib/kubelet/pods/2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf/volumes"
Sep 11 16:21:56.034518 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:56.034503    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a143c913-c93b-459d-bf1b-1cce98e8323f" path="/var/lib/kubelet/pods/a143c913-c93b-459d-bf1b-1cce98e8323f/volumes"
Sep 11 16:21:56.034797 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:21:56.034785    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad18e40b-ad28-4a3c-9235-9a6dae240b8c" path="/var/lib/kubelet/pods/ad18e40b-ad28-4a3c-9235-9a6dae240b8c/volumes"
Sep 11 16:22:04.248285 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:04.248251    2559 scope.go:117] "RemoveContainer" containerID="80e3fde2b47262af6df81aaaf4eb590b3a9af11839d0bfcf32ccef93288d2ff8"
Sep 11 16:22:04.257194 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:04.257172    2559 scope.go:117] "RemoveContainer" containerID="2c7f9bcbda46bcf0f131b30bdb34ad6403b1705a2601e886c6540db3c0dfe10e"
Sep 11 16:22:04.265341 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:04.265291    2559 scope.go:117] "RemoveContainer" containerID="286ff6c03b53c937dacd10f5caa16c68260a1e02f33c5b56f730da2d63b8c8bd"
Sep 11 16:22:12.178596 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.178561    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"]
Sep 11 16:22:12.179069 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.178777    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr" podUID="c2ae7293-78f3-4f12-926e-e7bb546d350b" containerName="payload-processing" containerID="cri-o://76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9" gracePeriod=30
Sep 11 16:22:12.189541 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.189514    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"]
Sep 11 16:22:12.190062 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.189743    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9" podUID="9fe37b99-eec2-433f-9913-5159e5acb8b2" containerName="payload-pre-processing" containerID="cri-o://1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe" gracePeriod=30
Sep 11 16:22:12.461104 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.461081    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:22:12.464445 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.464425    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:22:12.601435 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.601398    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ae7293-78f3-4f12-926e-e7bb546d350b-plugins-config-volume\") pod \"c2ae7293-78f3-4f12-926e-e7bb546d350b\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") "
Sep 11 16:22:12.601605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.601474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsch7\" (UniqueName: \"kubernetes.io/projected/c2ae7293-78f3-4f12-926e-e7bb546d350b-kube-api-access-hsch7\") pod \"c2ae7293-78f3-4f12-926e-e7bb546d350b\" (UID: \"c2ae7293-78f3-4f12-926e-e7bb546d350b\") "
Sep 11 16:22:12.601605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.601528    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe37b99-eec2-433f-9913-5159e5acb8b2-plugins-config-volume\") pod \"9fe37b99-eec2-433f-9913-5159e5acb8b2\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") "
Sep 11 16:22:12.601605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.601598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxq4d\" (UniqueName: \"kubernetes.io/projected/9fe37b99-eec2-433f-9913-5159e5acb8b2-kube-api-access-gxq4d\") pod \"9fe37b99-eec2-433f-9913-5159e5acb8b2\" (UID: \"9fe37b99-eec2-433f-9913-5159e5acb8b2\") "
Sep 11 16:22:12.601864 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.601836    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ae7293-78f3-4f12-926e-e7bb546d350b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2ae7293-78f3-4f12-926e-e7bb546d350b" (UID: "c2ae7293-78f3-4f12-926e-e7bb546d350b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:22:12.601938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.601904    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe37b99-eec2-433f-9913-5159e5acb8b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9fe37b99-eec2-433f-9913-5159e5acb8b2" (UID: "9fe37b99-eec2-433f-9913-5159e5acb8b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:22:12.603685 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.603657    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe37b99-eec2-433f-9913-5159e5acb8b2-kube-api-access-gxq4d" (OuterVolumeSpecName: "kube-api-access-gxq4d") pod "9fe37b99-eec2-433f-9913-5159e5acb8b2" (UID: "9fe37b99-eec2-433f-9913-5159e5acb8b2"). InnerVolumeSpecName "kube-api-access-gxq4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:22:12.603785 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.603687    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ae7293-78f3-4f12-926e-e7bb546d350b-kube-api-access-hsch7" (OuterVolumeSpecName: "kube-api-access-hsch7") pod "c2ae7293-78f3-4f12-926e-e7bb546d350b" (UID: "c2ae7293-78f3-4f12-926e-e7bb546d350b"). InnerVolumeSpecName "kube-api-access-hsch7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:22:12.703011 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.702938    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ae7293-78f3-4f12-926e-e7bb546d350b-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:12.703011 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.702966    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsch7\" (UniqueName: \"kubernetes.io/projected/c2ae7293-78f3-4f12-926e-e7bb546d350b-kube-api-access-hsch7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:12.703011 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.702976    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe37b99-eec2-433f-9913-5159e5acb8b2-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:12.703011 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:12.702984    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxq4d\" (UniqueName: \"kubernetes.io/projected/9fe37b99-eec2-433f-9913-5159e5acb8b2-kube-api-access-gxq4d\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:13.196172 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.196140    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fe37b99-eec2-433f-9913-5159e5acb8b2" containerID="1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe" exitCode=0
Sep 11 16:22:13.196614 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.196183    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"
Sep 11 16:22:13.196614 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.196252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9" event={"ID":"9fe37b99-eec2-433f-9913-5159e5acb8b2","Type":"ContainerDied","Data":"1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe"}
Sep 11 16:22:13.196614 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.196308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9" event={"ID":"9fe37b99-eec2-433f-9913-5159e5acb8b2","Type":"ContainerDied","Data":"eb6c2c3ee8f80aa7e9c5af09f431d8a4ee6bdfc04bc1aa1cb849309de76ba118"}
Sep 11 16:22:13.196614 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.196330    2559 scope.go:117] "RemoveContainer" containerID="1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe"
Sep 11 16:22:13.198417 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.198395    2559 generic.go:356] "Generic (PLEG): container finished" podID="c2ae7293-78f3-4f12-926e-e7bb546d350b" containerID="76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9" exitCode=0
Sep 11 16:22:13.198527 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.198430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr" event={"ID":"c2ae7293-78f3-4f12-926e-e7bb546d350b","Type":"ContainerDied","Data":"76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9"}
Sep 11 16:22:13.198527 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.198454    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"
Sep 11 16:22:13.198527 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.198466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr" event={"ID":"c2ae7293-78f3-4f12-926e-e7bb546d350b","Type":"ContainerDied","Data":"1a8aab57182b71f75d7cff012cd6f32ef1dd555fb339fa0bd04bcd1ac59cff46"}
Sep 11 16:22:13.207478 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.207395    2559 scope.go:117] "RemoveContainer" containerID="1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe"
Sep 11 16:22:13.207980 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:22:13.207955    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe\": container with ID starting with 1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe not found: ID does not exist" containerID="1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe"
Sep 11 16:22:13.208088 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.207989    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe"} err="failed to get container status \"1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe\": rpc error: code = NotFound desc = could not find container \"1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe\": container with ID starting with 1724eebb25d2e26eebebe4f1f38563f86bf3ca2fb180e7eda80b5bead5f1f9fe not found: ID does not exist"
Sep 11 16:22:13.208088 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.208013    2559 scope.go:117] "RemoveContainer" containerID="76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9"
Sep 11 16:22:13.215982 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.215965    2559 scope.go:117] "RemoveContainer" containerID="76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9"
Sep 11 16:22:13.216230 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:22:13.216200    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9\": container with ID starting with 76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9 not found: ID does not exist" containerID="76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9"
Sep 11 16:22:13.216300 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.216263    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9"} err="failed to get container status \"76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9\": rpc error: code = NotFound desc = could not find container \"76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9\": container with ID starting with 76f081e5d700fdbc5cb30a5165a5533fecd7df61cd8fe909570a5b2b6b8c1ae9 not found: ID does not exist"
Sep 11 16:22:13.243073 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.243052    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"]
Sep 11 16:22:13.249019 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.248999    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-320b99-7dc498d6cc-j9xgr"]
Sep 11 16:22:13.280014 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.279990    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"]
Sep 11 16:22:13.286343 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:13.286322    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-320b99-86dd9d7c58-bnzc9"]
Sep 11 16:22:14.034805 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:14.034769    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe37b99-eec2-433f-9913-5159e5acb8b2" path="/var/lib/kubelet/pods/9fe37b99-eec2-433f-9913-5159e5acb8b2/volumes"
Sep 11 16:22:14.035139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:14.035125    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ae7293-78f3-4f12-926e-e7bb546d350b" path="/var/lib/kubelet/pods/c2ae7293-78f3-4f12-926e-e7bb546d350b/volumes"
Sep 11 16:22:41.679066 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:41.679033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"]
Sep 11 16:22:50.686687 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.686652    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"]
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687016    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad18e40b-ad28-4a3c-9235-9a6dae240b8c" containerName="revoke-keys"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687027    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad18e40b-ad28-4a3c-9235-9a6dae240b8c" containerName="revoke-keys"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687037    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe37b99-eec2-433f-9913-5159e5acb8b2" containerName="payload-pre-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687043    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe37b99-eec2-433f-9913-5159e5acb8b2" containerName="payload-pre-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687060    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a143c913-c93b-459d-bf1b-1cce98e8323f" containerName="payload-pre-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687066    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a143c913-c93b-459d-bf1b-1cce98e8323f" containerName="payload-pre-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687077    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" containerName="payload-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687082    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" containerName="payload-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687088    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ae7293-78f3-4f12-926e-e7bb546d350b" containerName="payload-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687094    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ae7293-78f3-4f12-926e-e7bb546d350b" containerName="payload-processing"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687149    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad18e40b-ad28-4a3c-9235-9a6dae240b8c" containerName="revoke-keys"
Sep 11 16:22:50.687154 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687156    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe37b99-eec2-433f-9913-5159e5acb8b2" containerName="payload-pre-processing"
Sep 11 16:22:50.687552 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687164    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ae7293-78f3-4f12-926e-e7bb546d350b" containerName="payload-processing"
Sep 11 16:22:50.687552 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687174    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a143c913-c93b-459d-bf1b-1cce98e8323f" containerName="payload-pre-processing"
Sep 11 16:22:50.687552 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.687180    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b26f06c-21b5-4ed1-a2a6-48cde3ad23bf" containerName="payload-processing"
Sep 11 16:22:50.690402 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.690381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:50.693965 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.693946    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f9d303\""
Sep 11 16:22:50.703113 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.703086    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"]
Sep 11 16:22:50.754718 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.754689    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"]
Sep 11 16:22:50.756029 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.756005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:50.756155 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.756135    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqtd\" (UniqueName: \"kubernetes.io/projected/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-kube-api-access-sxqtd\") pod \"payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:50.758321 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.758307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:50.761164 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.761145    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f9d303-dockercfg-5hqkf\""
Sep 11 16:22:50.770916 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.770893    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"]
Sep 11 16:22:50.857261 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.857207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gmr\" (UniqueName: \"kubernetes.io/projected/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-kube-api-access-85gmr\") pod \"payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:50.857444 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.857271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqtd\" (UniqueName: \"kubernetes.io/projected/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-kube-api-access-sxqtd\") pod \"payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:50.857444 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.857322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:50.857444 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.857384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:50.858078 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.858053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:50.865989 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.865969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqtd\" (UniqueName: \"kubernetes.io/projected/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-kube-api-access-sxqtd\") pod \"payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:50.958545 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.958456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:50.958545 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.958527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gmr\" (UniqueName: \"kubernetes.io/projected/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-kube-api-access-85gmr\") pod \"payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:50.959191 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.959164    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:50.970194 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:50.970159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gmr\" (UniqueName: \"kubernetes.io/projected/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-kube-api-access-85gmr\") pod \"payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:51.000562 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.000532    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:51.068734 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.068702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:51.280629 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.280600    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"]
Sep 11 16:22:51.281462 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:22:51.281434    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfba37a_9b8e_4a25_a865_8f5a8b22d4ef.slice/crio-e319cbd7276ed71b00f8f0eaf3962dac7d6e4327443bcbb319e9426589c2ed88 WatchSource:0}: Error finding container e319cbd7276ed71b00f8f0eaf3962dac7d6e4327443bcbb319e9426589c2ed88: Status 404 returned error can't find the container with id e319cbd7276ed71b00f8f0eaf3962dac7d6e4327443bcbb319e9426589c2ed88
Sep 11 16:22:51.347067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.347042    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"]
Sep 11 16:22:51.351513 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:22:51.351488    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a59f7b5_6efe_40d3_b057_a0cfeddbcf9f.slice/crio-64cd8f52345798ccf685b6a9f7c384416a943ce6507586a6613bcce965a2aed1 WatchSource:0}: Error finding container 64cd8f52345798ccf685b6a9f7c384416a943ce6507586a6613bcce965a2aed1: Status 404 returned error can't find the container with id 64cd8f52345798ccf685b6a9f7c384416a943ce6507586a6613bcce965a2aed1
Sep 11 16:22:51.392010 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.391963    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs" event={"ID":"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef","Type":"ContainerStarted","Data":"6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb"}
Sep 11 16:22:51.392172 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.392146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:22:51.392263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.392196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs" event={"ID":"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef","Type":"ContainerStarted","Data":"e319cbd7276ed71b00f8f0eaf3962dac7d6e4327443bcbb319e9426589c2ed88"}
Sep 11 16:22:51.393401 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.393376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk" event={"ID":"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f","Type":"ContainerStarted","Data":"64cd8f52345798ccf685b6a9f7c384416a943ce6507586a6613bcce965a2aed1"}
Sep 11 16:22:51.414219 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:51.414161    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs" podStartSLOduration=1.414140334 podStartE2EDuration="1.414140334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:22:51.411564608 +0000 UTC m=+1667.835382330" watchObservedRunningTime="2026-09-11 16:22:51.414140334 +0000 UTC m=+1667.837958029"
Sep 11 16:22:52.400561 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:52.400521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk" event={"ID":"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f","Type":"ContainerStarted","Data":"be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479"}
Sep 11 16:22:52.400968 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:52.400599    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:22:52.428354 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:52.422269    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk" podStartSLOduration=2.422228487 podStartE2EDuration="2.422228487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:22:52.418791048 +0000 UTC m=+1668.842608763" watchObservedRunningTime="2026-09-11 16:22:52.422228487 +0000 UTC m=+1668.846046182"
Sep 11 16:22:56.697825 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:56.697784    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w" podUID="a4a464eb-3152-4d67-bc99-b22de0089e2d" containerName="main" containerID="cri-o://67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e" gracePeriod=45
Sep 11 16:22:56.955941 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:56.955879    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:22:57.115662 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.115625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s42wf\" (UniqueName: \"kubernetes.io/projected/a4a464eb-3152-4d67-bc99-b22de0089e2d-kube-api-access-s42wf\") pod \"a4a464eb-3152-4d67-bc99-b22de0089e2d\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") "
Sep 11 16:22:57.115871 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.115691    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-home\") pod \"a4a464eb-3152-4d67-bc99-b22de0089e2d\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") "
Sep 11 16:22:57.115871 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.115717    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-tmp-dir\") pod \"a4a464eb-3152-4d67-bc99-b22de0089e2d\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") "
Sep 11 16:22:57.115871 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.115744    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-dshm\") pod \"a4a464eb-3152-4d67-bc99-b22de0089e2d\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") "
Sep 11 16:22:57.115871 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.115782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-model-cache\") pod \"a4a464eb-3152-4d67-bc99-b22de0089e2d\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") "
Sep 11 16:22:57.115871 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.115814    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4a464eb-3152-4d67-bc99-b22de0089e2d-tls-certs\") pod \"a4a464eb-3152-4d67-bc99-b22de0089e2d\" (UID: \"a4a464eb-3152-4d67-bc99-b22de0089e2d\") "
Sep 11 16:22:57.116126 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.116029    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-home" (OuterVolumeSpecName: "home") pod "a4a464eb-3152-4d67-bc99-b22de0089e2d" (UID: "a4a464eb-3152-4d67-bc99-b22de0089e2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.116126 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.116057    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4a464eb-3152-4d67-bc99-b22de0089e2d" (UID: "a4a464eb-3152-4d67-bc99-b22de0089e2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.116126 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.116068    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4a464eb-3152-4d67-bc99-b22de0089e2d" (UID: "a4a464eb-3152-4d67-bc99-b22de0089e2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.117919 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.117885    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-dshm" (OuterVolumeSpecName: "dshm") pod "a4a464eb-3152-4d67-bc99-b22de0089e2d" (UID: "a4a464eb-3152-4d67-bc99-b22de0089e2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.118025 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.117924    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a464eb-3152-4d67-bc99-b22de0089e2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4a464eb-3152-4d67-bc99-b22de0089e2d" (UID: "a4a464eb-3152-4d67-bc99-b22de0089e2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:22:57.118025 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.117927    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a464eb-3152-4d67-bc99-b22de0089e2d-kube-api-access-s42wf" (OuterVolumeSpecName: "kube-api-access-s42wf") pod "a4a464eb-3152-4d67-bc99-b22de0089e2d" (UID: "a4a464eb-3152-4d67-bc99-b22de0089e2d"). InnerVolumeSpecName "kube-api-access-s42wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:22:57.217263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.217174    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42wf\" (UniqueName: \"kubernetes.io/projected/a4a464eb-3152-4d67-bc99-b22de0089e2d-kube-api-access-s42wf\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.217263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.217200    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.217263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.217212    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.217263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.217254    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.217263 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.217263    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4a464eb-3152-4d67-bc99-b22de0089e2d-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.217511 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.217271    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4a464eb-3152-4d67-bc99-b22de0089e2d-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.426372 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.426339    2559 generic.go:356] "Generic (PLEG): container finished" podID="a4a464eb-3152-4d67-bc99-b22de0089e2d" containerID="67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e" exitCode=0
Sep 11 16:22:57.426531 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.426385    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"
Sep 11 16:22:57.426531 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.426427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w" event={"ID":"a4a464eb-3152-4d67-bc99-b22de0089e2d","Type":"ContainerDied","Data":"67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e"}
Sep 11 16:22:57.426531 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.426470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w" event={"ID":"a4a464eb-3152-4d67-bc99-b22de0089e2d","Type":"ContainerDied","Data":"dbaaa21f18e5ee4068372051098548c6f98a8f469bd995811f1697f7bb93f466"}
Sep 11 16:22:57.426531 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.426492    2559 scope.go:117] "RemoveContainer" containerID="67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e"
Sep 11 16:22:57.435401 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.435382    2559 scope.go:117] "RemoveContainer" containerID="67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e"
Sep 11 16:22:57.435658 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:22:57.435639    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e\": container with ID starting with 67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e not found: ID does not exist" containerID="67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e"
Sep 11 16:22:57.435728 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.435669    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e"} err="failed to get container status \"67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e\": rpc error: code = NotFound desc = could not find container \"67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e\": container with ID starting with 67906705bf5a7a3ff4eefca10291fb26f07d3eb5300e1ca26ec7974c4d83659e not found: ID does not exist"
Sep 11 16:22:57.451059 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.451036    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"]
Sep 11 16:22:57.453592 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:57.453570    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/rate-test-model-59f108-kserve-6cd65b999f-prg4w"]
Sep 11 16:22:58.034794 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:22:58.034751    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a464eb-3152-4d67-bc99-b22de0089e2d" path="/var/lib/kubelet/pods/a4a464eb-3152-4d67-bc99-b22de0089e2d/volumes"
Sep 11 16:23:02.402970 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:02.402932    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:23:03.408373 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:03.408344    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:23:27.031621 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.031549    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"]
Sep 11 16:23:27.032056 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.031925    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a464eb-3152-4d67-bc99-b22de0089e2d" containerName="main"
Sep 11 16:23:27.032056 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.031935    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a464eb-3152-4d67-bc99-b22de0089e2d" containerName="main"
Sep 11 16:23:27.032056 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.032005    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a464eb-3152-4d67-bc99-b22de0089e2d" containerName="main"
Sep 11 16:23:27.035360 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.035345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.038678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.038656    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c0aa4c-dockercfg-7nvxr\""
Sep 11 16:23:27.038993 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.038979    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c0aa4c\""
Sep 11 16:23:27.050019 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.049996    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"]
Sep 11 16:23:27.053463 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.053436    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.061056 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.061032    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"]
Sep 11 16:23:27.093082 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.093054    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"]
Sep 11 16:23:27.093276 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.093101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b768037-bb48-48aa-99ee-4f94097e3d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.093358 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.093296    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8t9d\" (UniqueName: \"kubernetes.io/projected/1b768037-bb48-48aa-99ee-4f94097e3d8a-kube-api-access-k8t9d\") pod \"payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.194002 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.193964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8t9d\" (UniqueName: \"kubernetes.io/projected/1b768037-bb48-48aa-99ee-4f94097e3d8a-kube-api-access-k8t9d\") pod \"payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.194193 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.194010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a775e-e079-449c-96b7-b7551ecbd7b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.194193 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.194049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b768037-bb48-48aa-99ee-4f94097e3d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.194193 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.194074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5tr\" (UniqueName: \"kubernetes.io/projected/e77a775e-e079-449c-96b7-b7551ecbd7b9-kube-api-access-ns5tr\") pod \"payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.194821 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.194801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b768037-bb48-48aa-99ee-4f94097e3d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.210908 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.210875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8t9d\" (UniqueName: \"kubernetes.io/projected/1b768037-bb48-48aa-99ee-4f94097e3d8a-kube-api-access-k8t9d\") pod \"payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.294920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.294834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a775e-e079-449c-96b7-b7551ecbd7b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.294920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.294887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5tr\" (UniqueName: \"kubernetes.io/projected/e77a775e-e079-449c-96b7-b7551ecbd7b9-kube-api-access-ns5tr\") pod \"payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.295614 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.295587    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a775e-e079-449c-96b7-b7551ecbd7b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.306475 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.306450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5tr\" (UniqueName: \"kubernetes.io/projected/e77a775e-e079-449c-96b7-b7551ecbd7b9-kube-api-access-ns5tr\") pod \"payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.345739 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.345713    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:27.364023 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.363991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:27.628312 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.628275    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"]
Sep 11 16:23:27.629930 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:23:27.629903    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b768037_bb48_48aa_99ee_4f94097e3d8a.slice/crio-469f1cf67c05ef004f12788f8496f6e2efd1a739821380f2940a6c0e5fdecded WatchSource:0}: Error finding container 469f1cf67c05ef004f12788f8496f6e2efd1a739821380f2940a6c0e5fdecded: Status 404 returned error can't find the container with id 469f1cf67c05ef004f12788f8496f6e2efd1a739821380f2940a6c0e5fdecded
Sep 11 16:23:27.656037 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:27.656012    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"]
Sep 11 16:23:27.660393 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:23:27.660366    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77a775e_e079_449c_96b7_b7551ecbd7b9.slice/crio-2fa7b60945eaeb439e6aba50a607169f4f6a9ef1fbd5710c237d417ba5bd14ab WatchSource:0}: Error finding container 2fa7b60945eaeb439e6aba50a607169f4f6a9ef1fbd5710c237d417ba5bd14ab: Status 404 returned error can't find the container with id 2fa7b60945eaeb439e6aba50a607169f4f6a9ef1fbd5710c237d417ba5bd14ab
Sep 11 16:23:28.585813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.585776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s" event={"ID":"e77a775e-e079-449c-96b7-b7551ecbd7b9","Type":"ContainerStarted","Data":"65b36781b205628ec7ba5eecabef90cabbb395033ce8a137bd30b17138b30a47"}
Sep 11 16:23:28.586300 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.585948    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:28.586300 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.585991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s" event={"ID":"e77a775e-e079-449c-96b7-b7551ecbd7b9","Type":"ContainerStarted","Data":"2fa7b60945eaeb439e6aba50a607169f4f6a9ef1fbd5710c237d417ba5bd14ab"}
Sep 11 16:23:28.587513 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.587489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh" event={"ID":"1b768037-bb48-48aa-99ee-4f94097e3d8a","Type":"ContainerStarted","Data":"9d2c7daf048bb56d0e9808a33c057d6fa9435549c3acc52b0b8d2906127e4165"}
Sep 11 16:23:28.587665 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.587528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh" event={"ID":"1b768037-bb48-48aa-99ee-4f94097e3d8a","Type":"ContainerStarted","Data":"469f1cf67c05ef004f12788f8496f6e2efd1a739821380f2940a6c0e5fdecded"}
Sep 11 16:23:28.587665 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.587551    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:28.637428 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.637377    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s" podStartSLOduration=2.637362461 podStartE2EDuration="2.637362461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:28.636272582 +0000 UTC m=+1705.060090293" watchObservedRunningTime="2026-09-11 16:23:28.637362461 +0000 UTC m=+1705.061180213"
Sep 11 16:23:28.665491 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:28.665448    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh" podStartSLOduration=2.665432482 podStartE2EDuration="2.665432482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:28.663848108 +0000 UTC m=+1705.087665802" watchObservedRunningTime="2026-09-11 16:23:28.665432482 +0000 UTC m=+1705.089250176"
Sep 11 16:23:39.594342 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:39.594307    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:23:39.594760 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:39.594380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:23:56.469406 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:23:56.469373    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s_e77a775e-e079-449c-96b7-b7551ecbd7b9/payload-processing/0.log"
Sep 11 16:24:19.390374 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.390334    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"]
Sep 11 16:24:19.393766 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.393748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.401299 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.401278    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f9d303\"/\"openshift-service-ca.crt\""
Sep 11 16:24:19.401433 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.401390    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f9d303\"/\"default-dockercfg-nd5dx\""
Sep 11 16:24:19.401491 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.401431    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f9d303\"/\"kube-root-ca.crt\""
Sep 11 16:24:19.412074 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.412041    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f9d303\"/\"ipp-route-f9d303-kserve-self-signed-certs\""
Sep 11 16:24:19.414069 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.414034    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"]
Sep 11 16:24:19.490989 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.490949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tls-certs\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.490989 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.490995    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-model-cache\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.491314 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.491020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8nb\" (UniqueName: \"kubernetes.io/projected/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-kube-api-access-jc8nb\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.491314 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.491154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tmp-dir\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.491314 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.491210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-home\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.491314 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.491275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-dshm\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.591855 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.591793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tmp-dir\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592054 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.591934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-home\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592054 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.591974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-dshm\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592196 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.592107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tls-certs\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592196 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.592177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-model-cache\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592196 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.592182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tmp-dir\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592401 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.592213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8nb\" (UniqueName: \"kubernetes.io/projected/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-kube-api-access-jc8nb\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592401 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.592357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-home\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.592521 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.592499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-model-cache\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.594356 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.594327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-dshm\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.594813 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.594791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tls-certs\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.607889 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.607864    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8nb\" (UniqueName: \"kubernetes.io/projected/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-kube-api-access-jc8nb\") pod \"ipp-route-f9d303-kserve-57dcdff579-ll7rx\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") " pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.703230 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.703133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:19.995432 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:19.995398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"]
Sep 11 16:24:19.997213 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:24:19.997189    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ea88a4_524d_4dbb_8d18_b4f471dc1dc2.slice/crio-c46c6d94ecf732f3e880dd4f314e71fc36df6efe70951b8e822140de3d37ff3b WatchSource:0}: Error finding container c46c6d94ecf732f3e880dd4f314e71fc36df6efe70951b8e822140de3d37ff3b: Status 404 returned error can't find the container with id c46c6d94ecf732f3e880dd4f314e71fc36df6efe70951b8e822140de3d37ff3b
Sep 11 16:24:20.847259 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:20.847201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx" event={"ID":"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2","Type":"ContainerStarted","Data":"838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da"}
Sep 11 16:24:20.847675 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:20.847273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx" event={"ID":"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2","Type":"ContainerStarted","Data":"c46c6d94ecf732f3e880dd4f314e71fc36df6efe70951b8e822140de3d37ff3b"}
Sep 11 16:24:20.876579 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:20.876524    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx" podStartSLOduration=1.876506175 podStartE2EDuration="1.876506175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:20.870636099 +0000 UTC m=+1757.294453793" watchObservedRunningTime="2026-09-11 16:24:20.876506175 +0000 UTC m=+1757.300323868"
Sep 11 16:24:29.704170 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:29.704110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:29.704170 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:29.704180    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:29.717098 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:29.717067    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:29.904728 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:29.904700    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:24:42.574850 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:42.574817    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs_6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef/payload-processing/0.log"
Sep 11 16:24:42.873057 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:42.872979    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s_e77a775e-e079-449c-96b7-b7551ecbd7b9/payload-processing/0.log"
Sep 11 16:24:52.649067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.649032    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"]
Sep 11 16:24:52.652976 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.652957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:52.655971 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.655941    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-214daa\""
Sep 11 16:24:52.656083 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.656027    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-214daa-dockercfg-f8w4p\""
Sep 11 16:24:52.665361 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.665337    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"]
Sep 11 16:24:52.830321 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.830270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73133cb-1eec-4239-90f4-8c17d2206e68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:52.830521 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.830406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r575g\" (UniqueName: \"kubernetes.io/projected/d73133cb-1eec-4239-90f4-8c17d2206e68-kube-api-access-r575g\") pod \"payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:52.932048 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.931947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73133cb-1eec-4239-90f4-8c17d2206e68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:52.932199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.932067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r575g\" (UniqueName: \"kubernetes.io/projected/d73133cb-1eec-4239-90f4-8c17d2206e68-kube-api-access-r575g\") pod \"payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:52.932734 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.932711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73133cb-1eec-4239-90f4-8c17d2206e68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:52.946327 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.946297    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r575g\" (UniqueName: \"kubernetes.io/projected/d73133cb-1eec-4239-90f4-8c17d2206e68-kube-api-access-r575g\") pod \"payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:52.964722 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:52.964687    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:53.249074 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:53.249043    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"]
Sep 11 16:24:53.251274 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:24:53.251228    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73133cb_1eec_4239_90f4_8c17d2206e68.slice/crio-e3b438f02705a8c675587fbb6a8a84b6188dfb8ba57fbf58b92c43aea54ead4c WatchSource:0}: Error finding container e3b438f02705a8c675587fbb6a8a84b6188dfb8ba57fbf58b92c43aea54ead4c: Status 404 returned error can't find the container with id e3b438f02705a8c675587fbb6a8a84b6188dfb8ba57fbf58b92c43aea54ead4c
Sep 11 16:24:54.018556 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:54.018517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67" event={"ID":"d73133cb-1eec-4239-90f4-8c17d2206e68","Type":"ContainerStarted","Data":"e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31"}
Sep 11 16:24:54.018955 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:54.018579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67" event={"ID":"d73133cb-1eec-4239-90f4-8c17d2206e68","Type":"ContainerStarted","Data":"e3b438f02705a8c675587fbb6a8a84b6188dfb8ba57fbf58b92c43aea54ead4c"}
Sep 11 16:24:54.018955 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:54.018620    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:24:54.048645 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:24:54.048577    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67" podStartSLOduration=2.048555068 podStartE2EDuration="2.048555068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:54.045584495 +0000 UTC m=+1790.469402214" watchObservedRunningTime="2026-09-11 16:24:54.048555068 +0000 UTC m=+1790.472372764"
Sep 11 16:25:05.026734 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:05.026701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:25:28.794029 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:28.793994    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"]
Sep 11 16:25:28.794619 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:28.794219    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67" podUID="d73133cb-1eec-4239-90f4-8c17d2206e68" containerName="payload-pre-processing" containerID="cri-o://e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31" gracePeriod=30
Sep 11 16:25:29.056885 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.056816    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:25:29.170609 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.170575    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r575g\" (UniqueName: \"kubernetes.io/projected/d73133cb-1eec-4239-90f4-8c17d2206e68-kube-api-access-r575g\") pod \"d73133cb-1eec-4239-90f4-8c17d2206e68\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") "
Sep 11 16:25:29.170808 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.170620    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73133cb-1eec-4239-90f4-8c17d2206e68-plugins-config-volume\") pod \"d73133cb-1eec-4239-90f4-8c17d2206e68\" (UID: \"d73133cb-1eec-4239-90f4-8c17d2206e68\") "
Sep 11 16:25:29.171071 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.171036    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73133cb-1eec-4239-90f4-8c17d2206e68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d73133cb-1eec-4239-90f4-8c17d2206e68" (UID: "d73133cb-1eec-4239-90f4-8c17d2206e68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:29.172923 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.172898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73133cb-1eec-4239-90f4-8c17d2206e68-kube-api-access-r575g" (OuterVolumeSpecName: "kube-api-access-r575g") pod "d73133cb-1eec-4239-90f4-8c17d2206e68" (UID: "d73133cb-1eec-4239-90f4-8c17d2206e68"). InnerVolumeSpecName "kube-api-access-r575g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:29.201300 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.201262    2559 generic.go:356] "Generic (PLEG): container finished" podID="d73133cb-1eec-4239-90f4-8c17d2206e68" containerID="e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31" exitCode=0
Sep 11 16:25:29.201496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.201313    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"
Sep 11 16:25:29.201496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.201344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67" event={"ID":"d73133cb-1eec-4239-90f4-8c17d2206e68","Type":"ContainerDied","Data":"e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31"}
Sep 11 16:25:29.201496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.201408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67" event={"ID":"d73133cb-1eec-4239-90f4-8c17d2206e68","Type":"ContainerDied","Data":"e3b438f02705a8c675587fbb6a8a84b6188dfb8ba57fbf58b92c43aea54ead4c"}
Sep 11 16:25:29.201496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.201440    2559 scope.go:117] "RemoveContainer" containerID="e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31"
Sep 11 16:25:29.210694 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.210671    2559 scope.go:117] "RemoveContainer" containerID="e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31"
Sep 11 16:25:29.210998 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:25:29.210977    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31\": container with ID starting with e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31 not found: ID does not exist" containerID="e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31"
Sep 11 16:25:29.211058 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.211007    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31"} err="failed to get container status \"e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31\": rpc error: code = NotFound desc = could not find container \"e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31\": container with ID starting with e9a1038b728283feb3f0f416617b9fd2977e5dfa4c80607e1744f8eaed4a5c31 not found: ID does not exist"
Sep 11 16:25:29.229304 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.229270    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"]
Sep 11 16:25:29.239074 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.239043    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-214daa-54bc5975b6-ghp67"]
Sep 11 16:25:29.272165 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.272128    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r575g\" (UniqueName: \"kubernetes.io/projected/d73133cb-1eec-4239-90f4-8c17d2206e68-kube-api-access-r575g\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:25:29.272165 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:29.272155    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73133cb-1eec-4239-90f4-8c17d2206e68-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:25:30.034812 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:30.034772    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73133cb-1eec-4239-90f4-8c17d2206e68" path="/var/lib/kubelet/pods/d73133cb-1eec-4239-90f4-8c17d2206e68/volumes"
Sep 11 16:25:55.407671 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.407631    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"]
Sep 11 16:25:55.410088 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.407958    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs" podUID="6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" containerName="payload-processing" containerID="cri-o://6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb" gracePeriod=30
Sep 11 16:25:55.450490 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.450448    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"]
Sep 11 16:25:55.450696 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.450658    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk" podUID="1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" containerName="payload-pre-processing" containerID="cri-o://be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479" gracePeriod=30
Sep 11 16:25:55.730694 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.730668    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:25:55.733706 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.733684    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:25:55.819544 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.819502    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85gmr\" (UniqueName: \"kubernetes.io/projected/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-kube-api-access-85gmr\") pod \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") "
Sep 11 16:25:55.819753 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.819555    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-plugins-config-volume\") pod \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") "
Sep 11 16:25:55.819753 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.819646    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxqtd\" (UniqueName: \"kubernetes.io/projected/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-kube-api-access-sxqtd\") pod \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\" (UID: \"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef\") "
Sep 11 16:25:55.819893 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.819821    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-plugins-config-volume\") pod \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\" (UID: \"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f\") "
Sep 11 16:25:55.819988 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.819954    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" (UID: "6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:55.820112 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.820088    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" (UID: "1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:55.820302 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.820282    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:25:55.820350 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.820307    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:25:55.821831 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.821803    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-kube-api-access-sxqtd" (OuterVolumeSpecName: "kube-api-access-sxqtd") pod "6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" (UID: "6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef"). InnerVolumeSpecName "kube-api-access-sxqtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:55.821831 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.821815    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-kube-api-access-85gmr" (OuterVolumeSpecName: "kube-api-access-85gmr") pod "1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" (UID: "1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f"). InnerVolumeSpecName "kube-api-access-85gmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:55.921658 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.921615    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxqtd\" (UniqueName: \"kubernetes.io/projected/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef-kube-api-access-sxqtd\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:25:55.921658 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:55.921651    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85gmr\" (UniqueName: \"kubernetes.io/projected/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f-kube-api-access-85gmr\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:25:56.340718 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.340621    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" containerID="6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb" exitCode=0
Sep 11 16:25:56.340718 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.340665    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"
Sep 11 16:25:56.340978 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.340710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs" event={"ID":"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef","Type":"ContainerDied","Data":"6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb"}
Sep 11 16:25:56.340978 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.340769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs" event={"ID":"6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef","Type":"ContainerDied","Data":"e319cbd7276ed71b00f8f0eaf3962dac7d6e4327443bcbb319e9426589c2ed88"}
Sep 11 16:25:56.340978 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.340798    2559 scope.go:117] "RemoveContainer" containerID="6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb"
Sep 11 16:25:56.342191 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.342170    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" containerID="be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479" exitCode=0
Sep 11 16:25:56.342313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.342209    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"
Sep 11 16:25:56.342313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.342210    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk" event={"ID":"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f","Type":"ContainerDied","Data":"be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479"}
Sep 11 16:25:56.342437 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.342328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk" event={"ID":"1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f","Type":"ContainerDied","Data":"64cd8f52345798ccf685b6a9f7c384416a943ce6507586a6613bcce965a2aed1"}
Sep 11 16:25:56.352415 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.352389    2559 scope.go:117] "RemoveContainer" containerID="6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb"
Sep 11 16:25:56.352767 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:25:56.352745    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb\": container with ID starting with 6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb not found: ID does not exist" containerID="6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb"
Sep 11 16:25:56.352858 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.352780    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb"} err="failed to get container status \"6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb\": rpc error: code = NotFound desc = could not find container \"6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb\": container with ID starting with 6065427eca7f7a9e8b4c6413533cfd52346ffc2c9996899bbd87592c7adf3bfb not found: ID does not exist"
Sep 11 16:25:56.352858 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.352801    2559 scope.go:117] "RemoveContainer" containerID="be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479"
Sep 11 16:25:56.361098 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.361065    2559 scope.go:117] "RemoveContainer" containerID="be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479"
Sep 11 16:25:56.361404 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:25:56.361379    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479\": container with ID starting with be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479 not found: ID does not exist" containerID="be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479"
Sep 11 16:25:56.361507 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.361412    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479"} err="failed to get container status \"be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479\": rpc error: code = NotFound desc = could not find container \"be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479\": container with ID starting with be213261d9e40f79a75deca03a56a070f7e5d5929c8e2fc85e7779b3bff0f479 not found: ID does not exist"
Sep 11 16:25:56.381013 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.380979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"]
Sep 11 16:25:56.386748 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.386712    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9d303-6f66fb9cbc-lnvvs"]
Sep 11 16:25:56.418930 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.418895    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"]
Sep 11 16:25:56.432416 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:56.432385    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f9d303-6f758d5f58-mvkfk"]
Sep 11 16:25:58.034221 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:58.034188    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" path="/var/lib/kubelet/pods/1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f/volumes"
Sep 11 16:25:58.034623 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:25:58.034607    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" path="/var/lib/kubelet/pods/6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef/volumes"
Sep 11 16:26:01.097370 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:01.097328    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"]
Sep 11 16:26:16.116808 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.116754    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx" podUID="24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" containerName="main" containerID="cri-o://838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da" gracePeriod=45
Sep 11 16:26:16.372374 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.372305    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:26:16.444286 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.444255    2559 generic.go:356] "Generic (PLEG): container finished" podID="24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" containerID="838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da" exitCode=0
Sep 11 16:26:16.444451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.444305    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"
Sep 11 16:26:16.444451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.444338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx" event={"ID":"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2","Type":"ContainerDied","Data":"838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da"}
Sep 11 16:26:16.444451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.444383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx" event={"ID":"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2","Type":"ContainerDied","Data":"c46c6d94ecf732f3e880dd4f314e71fc36df6efe70951b8e822140de3d37ff3b"}
Sep 11 16:26:16.444451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.444400    2559 scope.go:117] "RemoveContainer" containerID="838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da"
Sep 11 16:26:16.453313 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.453297    2559 scope.go:117] "RemoveContainer" containerID="838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da"
Sep 11 16:26:16.453621 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:26:16.453600    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da\": container with ID starting with 838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da not found: ID does not exist" containerID="838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da"
Sep 11 16:26:16.453711 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.453631    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da"} err="failed to get container status \"838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da\": rpc error: code = NotFound desc = could not find container \"838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da\": container with ID starting with 838bc287f4103e010fc6d3fc887018c99876557c95cc76b8a22349b63744f4da not found: ID does not exist"
Sep 11 16:26:16.518132 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518093    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc8nb\" (UniqueName: \"kubernetes.io/projected/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-kube-api-access-jc8nb\") pod \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") "
Sep 11 16:26:16.518323 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-home\") pod \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") "
Sep 11 16:26:16.518323 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518275    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tls-certs\") pod \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") "
Sep 11 16:26:16.518323 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518309    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tmp-dir\") pod \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") "
Sep 11 16:26:16.518451 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518355    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-dshm\") pod \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") "
Sep 11 16:26:16.518496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-home" (OuterVolumeSpecName: "home") pod "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" (UID: "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:16.518544 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518507    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-model-cache\") pod \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\" (UID: \"24ea88a4-524d-4dbb-8d18-b4f471dc1dc2\") "
Sep 11 16:26:16.518655 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518610    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" (UID: "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:16.518781 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518764    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" (UID: "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:16.518912 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518892    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:16.518983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518920    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:16.518983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.518936    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:16.520523 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.520502    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-dshm" (OuterVolumeSpecName: "dshm") pod "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" (UID: "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:16.520618 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.520532    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" (UID: "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:26:16.520618 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.520586    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-kube-api-access-jc8nb" (OuterVolumeSpecName: "kube-api-access-jc8nb") pod "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" (UID: "24ea88a4-524d-4dbb-8d18-b4f471dc1dc2"). InnerVolumeSpecName "kube-api-access-jc8nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:26:16.619691 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.619652    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc8nb\" (UniqueName: \"kubernetes.io/projected/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-kube-api-access-jc8nb\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:16.619691 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.619685    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:16.619691 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.619695    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:16.768750 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.768717    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"]
Sep 11 16:26:16.771821 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:16.771794    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f9d303/ipp-route-f9d303-kserve-57dcdff579-ll7rx"]
Sep 11 16:26:18.034473 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:18.034431    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" path="/var/lib/kubelet/pods/24ea88a4-524d-4dbb-8d18-b4f471dc1dc2/volumes"
Sep 11 16:26:44.786502 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786466    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"]
Sep 11 16:26:44.786889 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786881    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73133cb-1eec-4239-90f4-8c17d2206e68" containerName="payload-pre-processing"
Sep 11 16:26:44.786935 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786893    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73133cb-1eec-4239-90f4-8c17d2206e68" containerName="payload-pre-processing"
Sep 11 16:26:44.786935 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786901    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" containerName="payload-pre-processing"
Sep 11 16:26:44.786935 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786908    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" containerName="payload-pre-processing"
Sep 11 16:26:44.786935 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786915    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" containerName="payload-processing"
Sep 11 16:26:44.786935 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786921    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" containerName="payload-processing"
Sep 11 16:26:44.786935 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786933    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" containerName="main"
Sep 11 16:26:44.787203 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.786939    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" containerName="main"
Sep 11 16:26:44.787203 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.787001    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ea88a4-524d-4dbb-8d18-b4f471dc1dc2" containerName="main"
Sep 11 16:26:44.787203 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.787008    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a59f7b5-6efe-40d3-b057-a0cfeddbcf9f" containerName="payload-pre-processing"
Sep 11 16:26:44.787203 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.787014    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73133cb-1eec-4239-90f4-8c17d2206e68" containerName="payload-pre-processing"
Sep 11 16:26:44.787203 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.787021    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dfba37a-9b8e-4a25-a865-8f5a8b22d4ef" containerName="payload-processing"
Sep 11 16:26:44.789874 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.789856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"
Sep 11 16:26:44.793831 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.793809    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:26:44.801332 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.801306    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"]
Sep 11 16:26:44.881473 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.881419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f2105-f03f-4cda-8c3a-029ae7164800-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt\" (UID: \"365f2105-f03f-4cda-8c3a-029ae7164800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"
Sep 11 16:26:44.982141 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.982097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f2105-f03f-4cda-8c3a-029ae7164800-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt\" (UID: \"365f2105-f03f-4cda-8c3a-029ae7164800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"
Sep 11 16:26:44.982801 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:44.982773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f2105-f03f-4cda-8c3a-029ae7164800-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt\" (UID: \"365f2105-f03f-4cda-8c3a-029ae7164800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"
Sep 11 16:26:45.100495 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:45.100410    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"
Sep 11 16:26:45.374348 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:26:45.374315    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365f2105_f03f_4cda_8c3a_029ae7164800.slice/crio-3990bf59abe437c38be09777f7f2dff60ec3e764c9a0e76b34abb46f3e2ffec2 WatchSource:0}: Error finding container 3990bf59abe437c38be09777f7f2dff60ec3e764c9a0e76b34abb46f3e2ffec2: Status 404 returned error can't find the container with id 3990bf59abe437c38be09777f7f2dff60ec3e764c9a0e76b34abb46f3e2ffec2
Sep 11 16:26:45.374685 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:45.374658    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"]
Sep 11 16:26:45.592776 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:45.592739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt" event={"ID":"365f2105-f03f-4cda-8c3a-029ae7164800","Type":"ContainerStarted","Data":"3990bf59abe437c38be09777f7f2dff60ec3e764c9a0e76b34abb46f3e2ffec2"}
Sep 11 16:26:46.599028 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:46.598985    2559 generic.go:356] "Generic (PLEG): container finished" podID="365f2105-f03f-4cda-8c3a-029ae7164800" containerID="34561a71a2e92379957cc6821c7dfbd555a3322b8157e1094dc0b8e88ca5f962" exitCode=0
Sep 11 16:26:46.599438 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:46.599045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt" event={"ID":"365f2105-f03f-4cda-8c3a-029ae7164800","Type":"ContainerDied","Data":"34561a71a2e92379957cc6821c7dfbd555a3322b8157e1094dc0b8e88ca5f962"}
Sep 11 16:26:47.737302 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:47.737276    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"
Sep 11 16:26:47.807556 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:47.807515    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f2105-f03f-4cda-8c3a-029ae7164800-maas-api-service-ca\") pod \"365f2105-f03f-4cda-8c3a-029ae7164800\" (UID: \"365f2105-f03f-4cda-8c3a-029ae7164800\") "
Sep 11 16:26:47.807929 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:47.807900    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365f2105-f03f-4cda-8c3a-029ae7164800-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "365f2105-f03f-4cda-8c3a-029ae7164800" (UID: "365f2105-f03f-4cda-8c3a-029ae7164800"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:26:47.909077 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:47.908986    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365f2105-f03f-4cda-8c3a-029ae7164800-maas-api-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:48.609959 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:48.609917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt" event={"ID":"365f2105-f03f-4cda-8c3a-029ae7164800","Type":"ContainerDied","Data":"3990bf59abe437c38be09777f7f2dff60ec3e764c9a0e76b34abb46f3e2ffec2"}
Sep 11 16:26:48.610143 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:48.609979    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3990bf59abe437c38be09777f7f2dff60ec3e764c9a0e76b34abb46f3e2ffec2"
Sep 11 16:26:48.610143 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:48.609934    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"
Sep 11 16:26:55.500765 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.500722    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"]
Sep 11 16:26:55.501196 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.500967    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s" podUID="e77a775e-e079-449c-96b7-b7551ecbd7b9" containerName="payload-processing" containerID="cri-o://65b36781b205628ec7ba5eecabef90cabbb395033ce8a137bd30b17138b30a47" gracePeriod=30
Sep 11 16:26:55.507572 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.507538    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"]
Sep 11 16:26:55.507802 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.507771    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh" podUID="1b768037-bb48-48aa-99ee-4f94097e3d8a" containerName="payload-pre-processing" containerID="cri-o://9d2c7daf048bb56d0e9808a33c057d6fa9435549c3acc52b0b8d2906127e4165" gracePeriod=30
Sep 11 16:26:55.653298 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.653226    2559 generic.go:356] "Generic (PLEG): container finished" podID="e77a775e-e079-449c-96b7-b7551ecbd7b9" containerID="65b36781b205628ec7ba5eecabef90cabbb395033ce8a137bd30b17138b30a47" exitCode=0
Sep 11 16:26:55.653491 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.653265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s" event={"ID":"e77a775e-e079-449c-96b7-b7551ecbd7b9","Type":"ContainerDied","Data":"65b36781b205628ec7ba5eecabef90cabbb395033ce8a137bd30b17138b30a47"}
Sep 11 16:26:55.655418 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.655394    2559 generic.go:356] "Generic (PLEG): container finished" podID="1b768037-bb48-48aa-99ee-4f94097e3d8a" containerID="9d2c7daf048bb56d0e9808a33c057d6fa9435549c3acc52b0b8d2906127e4165" exitCode=0
Sep 11 16:26:55.655555 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.655469    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh" event={"ID":"1b768037-bb48-48aa-99ee-4f94097e3d8a","Type":"ContainerDied","Data":"9d2c7daf048bb56d0e9808a33c057d6fa9435549c3acc52b0b8d2906127e4165"}
Sep 11 16:26:55.715108 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.715051    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"]
Sep 11 16:26:55.718246 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.718208    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0aa4c-hr2wt"]
Sep 11 16:26:55.782835 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.782811    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:26:55.785532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.785513    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:26:55.899834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.899796    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8t9d\" (UniqueName: \"kubernetes.io/projected/1b768037-bb48-48aa-99ee-4f94097e3d8a-kube-api-access-k8t9d\") pod \"1b768037-bb48-48aa-99ee-4f94097e3d8a\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") "
Sep 11 16:26:55.899834 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.899850    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a775e-e079-449c-96b7-b7551ecbd7b9-plugins-config-volume\") pod \"e77a775e-e079-449c-96b7-b7551ecbd7b9\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") "
Sep 11 16:26:55.900067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.900002    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b768037-bb48-48aa-99ee-4f94097e3d8a-plugins-config-volume\") pod \"1b768037-bb48-48aa-99ee-4f94097e3d8a\" (UID: \"1b768037-bb48-48aa-99ee-4f94097e3d8a\") "
Sep 11 16:26:55.900067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.900054    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns5tr\" (UniqueName: \"kubernetes.io/projected/e77a775e-e079-449c-96b7-b7551ecbd7b9-kube-api-access-ns5tr\") pod \"e77a775e-e079-449c-96b7-b7551ecbd7b9\" (UID: \"e77a775e-e079-449c-96b7-b7551ecbd7b9\") "
Sep 11 16:26:55.900415 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.900383    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77a775e-e079-449c-96b7-b7551ecbd7b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e77a775e-e079-449c-96b7-b7551ecbd7b9" (UID: "e77a775e-e079-449c-96b7-b7551ecbd7b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:26:55.900415 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.900399    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b768037-bb48-48aa-99ee-4f94097e3d8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b768037-bb48-48aa-99ee-4f94097e3d8a" (UID: "1b768037-bb48-48aa-99ee-4f94097e3d8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:26:55.902073 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.902047    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b768037-bb48-48aa-99ee-4f94097e3d8a-kube-api-access-k8t9d" (OuterVolumeSpecName: "kube-api-access-k8t9d") pod "1b768037-bb48-48aa-99ee-4f94097e3d8a" (UID: "1b768037-bb48-48aa-99ee-4f94097e3d8a"). InnerVolumeSpecName "kube-api-access-k8t9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:26:55.902158 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:55.902099    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77a775e-e079-449c-96b7-b7551ecbd7b9-kube-api-access-ns5tr" (OuterVolumeSpecName: "kube-api-access-ns5tr") pod "e77a775e-e079-449c-96b7-b7551ecbd7b9" (UID: "e77a775e-e079-449c-96b7-b7551ecbd7b9"). InnerVolumeSpecName "kube-api-access-ns5tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:26:56.001552 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.001516    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b768037-bb48-48aa-99ee-4f94097e3d8a-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:56.001552 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.001545    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5tr\" (UniqueName: \"kubernetes.io/projected/e77a775e-e079-449c-96b7-b7551ecbd7b9-kube-api-access-ns5tr\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:56.001552 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.001554    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8t9d\" (UniqueName: \"kubernetes.io/projected/1b768037-bb48-48aa-99ee-4f94097e3d8a-kube-api-access-k8t9d\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:56.001552 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.001562    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a775e-e079-449c-96b7-b7551ecbd7b9-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:26:56.034817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.034733    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="365f2105-f03f-4cda-8c3a-029ae7164800" path="/var/lib/kubelet/pods/365f2105-f03f-4cda-8c3a-029ae7164800/volumes"
Sep 11 16:26:56.662161 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.662131    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"
Sep 11 16:26:56.662657 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.662130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh" event={"ID":"1b768037-bb48-48aa-99ee-4f94097e3d8a","Type":"ContainerDied","Data":"469f1cf67c05ef004f12788f8496f6e2efd1a739821380f2940a6c0e5fdecded"}
Sep 11 16:26:56.662657 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.662309    2559 scope.go:117] "RemoveContainer" containerID="9d2c7daf048bb56d0e9808a33c057d6fa9435549c3acc52b0b8d2906127e4165"
Sep 11 16:26:56.663884 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.663575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s" event={"ID":"e77a775e-e079-449c-96b7-b7551ecbd7b9","Type":"ContainerDied","Data":"2fa7b60945eaeb439e6aba50a607169f4f6a9ef1fbd5710c237d417ba5bd14ab"}
Sep 11 16:26:56.663884 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.663609    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"
Sep 11 16:26:56.671845 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.671825    2559 scope.go:117] "RemoveContainer" containerID="65b36781b205628ec7ba5eecabef90cabbb395033ce8a137bd30b17138b30a47"
Sep 11 16:26:56.680360 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.680330    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"]
Sep 11 16:26:56.684435 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.684405    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0aa4c-59c959c778-4q9jh"]
Sep 11 16:26:56.698452 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.698423    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"]
Sep 11 16:26:56.702948 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:56.702919    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0aa4c-765b8cd7d7-2775s"]
Sep 11 16:26:58.034294 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:58.034261    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b768037-bb48-48aa-99ee-4f94097e3d8a" path="/var/lib/kubelet/pods/1b768037-bb48-48aa-99ee-4f94097e3d8a/volumes"
Sep 11 16:26:58.034708 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:26:58.034692    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e77a775e-e079-449c-96b7-b7551ecbd7b9" path="/var/lib/kubelet/pods/e77a775e-e079-449c-96b7-b7551ecbd7b9/volumes"
Sep 11 16:27:03.012208 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012168    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"]
Sep 11 16:27:03.012598 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012564    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365f2105-f03f-4cda-8c3a-029ae7164800" containerName="revoke-keys"
Sep 11 16:27:03.012598 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012575    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="365f2105-f03f-4cda-8c3a-029ae7164800" containerName="revoke-keys"
Sep 11 16:27:03.012598 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012591    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77a775e-e079-449c-96b7-b7551ecbd7b9" containerName="payload-processing"
Sep 11 16:27:03.012598 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012597    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77a775e-e079-449c-96b7-b7551ecbd7b9" containerName="payload-processing"
Sep 11 16:27:03.012747 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012608    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b768037-bb48-48aa-99ee-4f94097e3d8a" containerName="payload-pre-processing"
Sep 11 16:27:03.012747 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012613    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b768037-bb48-48aa-99ee-4f94097e3d8a" containerName="payload-pre-processing"
Sep 11 16:27:03.012747 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012674    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b768037-bb48-48aa-99ee-4f94097e3d8a" containerName="payload-pre-processing"
Sep 11 16:27:03.012747 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012680    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77a775e-e079-449c-96b7-b7551ecbd7b9" containerName="payload-processing"
Sep 11 16:27:03.012747 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.012690    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="365f2105-f03f-4cda-8c3a-029ae7164800" containerName="revoke-keys"
Sep 11 16:27:03.017197 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.017175    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"
Sep 11 16:27:03.021986 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.021965    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:27:03.032561 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.032532    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"]
Sep 11 16:27:03.167441 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.167405    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/925b38b3-f013-4623-8050-2090b71ac95b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf\" (UID: \"925b38b3-f013-4623-8050-2090b71ac95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"
Sep 11 16:27:03.268886 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.268779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/925b38b3-f013-4623-8050-2090b71ac95b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf\" (UID: \"925b38b3-f013-4623-8050-2090b71ac95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"
Sep 11 16:27:03.269542 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.269518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/925b38b3-f013-4623-8050-2090b71ac95b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf\" (UID: \"925b38b3-f013-4623-8050-2090b71ac95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"
Sep 11 16:27:03.327627 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.327576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"
Sep 11 16:27:03.608094 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.607989    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"]
Sep 11 16:27:03.701259 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:03.701211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf" event={"ID":"925b38b3-f013-4623-8050-2090b71ac95b","Type":"ContainerStarted","Data":"5291628221d4ad22b7e420b948b4cff37f597d6bc100f55b81a0695a21a6eef5"}
Sep 11 16:27:04.707496 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:04.707401    2559 generic.go:356] "Generic (PLEG): container finished" podID="925b38b3-f013-4623-8050-2090b71ac95b" containerID="76bce976d1d21b29d45a53d6e87bc62d48c9ed98a9d3a6c2fa3056d968dfb7ea" exitCode=0
Sep 11 16:27:04.707857 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:04.707487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf" event={"ID":"925b38b3-f013-4623-8050-2090b71ac95b","Type":"ContainerDied","Data":"76bce976d1d21b29d45a53d6e87bc62d48c9ed98a9d3a6c2fa3056d968dfb7ea"}
Sep 11 16:27:05.846046 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:05.846018    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"
Sep 11 16:27:05.994070 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:05.993978    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/925b38b3-f013-4623-8050-2090b71ac95b-maas-api-service-ca\") pod \"925b38b3-f013-4623-8050-2090b71ac95b\" (UID: \"925b38b3-f013-4623-8050-2090b71ac95b\") "
Sep 11 16:27:05.994435 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:05.994412    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925b38b3-f013-4623-8050-2090b71ac95b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "925b38b3-f013-4623-8050-2090b71ac95b" (UID: "925b38b3-f013-4623-8050-2090b71ac95b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:06.095616 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:06.095580    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/925b38b3-f013-4623-8050-2090b71ac95b-maas-api-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:06.719668 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:06.719633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf" event={"ID":"925b38b3-f013-4623-8050-2090b71ac95b","Type":"ContainerDied","Data":"5291628221d4ad22b7e420b948b4cff37f597d6bc100f55b81a0695a21a6eef5"}
Sep 11 16:27:06.719846 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:06.719688    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5291628221d4ad22b7e420b948b4cff37f597d6bc100f55b81a0695a21a6eef5"
Sep 11 16:27:06.719846 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:06.719652    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"
Sep 11 16:27:13.717915 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.717875    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"]
Sep 11 16:27:13.718430 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.718130    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw" podUID="82dff3fe-b23b-4ca4-9552-45f9ca40a930" containerName="payload-processing" containerID="cri-o://caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1" gracePeriod=30
Sep 11 16:27:13.724969 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.724937    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"]
Sep 11 16:27:13.725249 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.725203    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp" podUID="56dff037-59a9-44d9-ad0e-fa2c61e3df8b" containerName="payload-pre-processing" containerID="cri-o://b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503" gracePeriod=30
Sep 11 16:27:13.921363 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.921254    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"]
Sep 11 16:27:13.926424 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.926319    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-59f108-5fwjf"]
Sep 11 16:27:13.993225 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.993199    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:27:13.996190 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:13.996168    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:27:14.043697 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.043653    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925b38b3-f013-4623-8050-2090b71ac95b" path="/var/lib/kubelet/pods/925b38b3-f013-4623-8050-2090b71ac95b/volumes"
Sep 11 16:27:14.071006 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.070967    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fmf4\" (UniqueName: \"kubernetes.io/projected/82dff3fe-b23b-4ca4-9552-45f9ca40a930-kube-api-access-8fmf4\") pod \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") "
Sep 11 16:27:14.071199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.071034    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84gxk\" (UniqueName: \"kubernetes.io/projected/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-kube-api-access-84gxk\") pod \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") "
Sep 11 16:27:14.071199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.071062    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-plugins-config-volume\") pod \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\" (UID: \"56dff037-59a9-44d9-ad0e-fa2c61e3df8b\") "
Sep 11 16:27:14.071199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.071179    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dff3fe-b23b-4ca4-9552-45f9ca40a930-plugins-config-volume\") pod \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\" (UID: \"82dff3fe-b23b-4ca4-9552-45f9ca40a930\") "
Sep 11 16:27:14.071563 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.071531    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56dff037-59a9-44d9-ad0e-fa2c61e3df8b" (UID: "56dff037-59a9-44d9-ad0e-fa2c61e3df8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:14.071683 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.071560    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dff3fe-b23b-4ca4-9552-45f9ca40a930-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82dff3fe-b23b-4ca4-9552-45f9ca40a930" (UID: "82dff3fe-b23b-4ca4-9552-45f9ca40a930"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:14.073277 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.073220    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dff3fe-b23b-4ca4-9552-45f9ca40a930-kube-api-access-8fmf4" (OuterVolumeSpecName: "kube-api-access-8fmf4") pod "82dff3fe-b23b-4ca4-9552-45f9ca40a930" (UID: "82dff3fe-b23b-4ca4-9552-45f9ca40a930"). InnerVolumeSpecName "kube-api-access-8fmf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:14.073367 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.073282    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-kube-api-access-84gxk" (OuterVolumeSpecName: "kube-api-access-84gxk") pod "56dff037-59a9-44d9-ad0e-fa2c61e3df8b" (UID: "56dff037-59a9-44d9-ad0e-fa2c61e3df8b"). InnerVolumeSpecName "kube-api-access-84gxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:14.172022 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.171974    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84gxk\" (UniqueName: \"kubernetes.io/projected/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-kube-api-access-84gxk\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.172022 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.172012    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dff037-59a9-44d9-ad0e-fa2c61e3df8b-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.172022 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.172026    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82dff3fe-b23b-4ca4-9552-45f9ca40a930-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.172286 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.172041    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fmf4\" (UniqueName: \"kubernetes.io/projected/82dff3fe-b23b-4ca4-9552-45f9ca40a930-kube-api-access-8fmf4\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.760626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.760583    2559 generic.go:356] "Generic (PLEG): container finished" podID="56dff037-59a9-44d9-ad0e-fa2c61e3df8b" containerID="b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503" exitCode=0
Sep 11 16:27:14.761291 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.760644    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"
Sep 11 16:27:14.761291 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.760672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp" event={"ID":"56dff037-59a9-44d9-ad0e-fa2c61e3df8b","Type":"ContainerDied","Data":"b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503"}
Sep 11 16:27:14.761291 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.760730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp" event={"ID":"56dff037-59a9-44d9-ad0e-fa2c61e3df8b","Type":"ContainerDied","Data":"287d12e7abe7f917d7e45a558ad06932ced984ad865853a6b718a5207806e24e"}
Sep 11 16:27:14.761291 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.760758    2559 scope.go:117] "RemoveContainer" containerID="b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503"
Sep 11 16:27:14.762449 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.762422    2559 generic.go:356] "Generic (PLEG): container finished" podID="82dff3fe-b23b-4ca4-9552-45f9ca40a930" containerID="caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1" exitCode=0
Sep 11 16:27:14.762567 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.762455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw" event={"ID":"82dff3fe-b23b-4ca4-9552-45f9ca40a930","Type":"ContainerDied","Data":"caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1"}
Sep 11 16:27:14.762567 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.762508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw" event={"ID":"82dff3fe-b23b-4ca4-9552-45f9ca40a930","Type":"ContainerDied","Data":"095716e896e1aa2ef8d177a1d5479acdf8490a62c388219ab720497715b969e2"}
Sep 11 16:27:14.762567 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.762521    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"
Sep 11 16:27:14.769676 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.769656    2559 scope.go:117] "RemoveContainer" containerID="b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503"
Sep 11 16:27:14.770018 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:27:14.769997    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503\": container with ID starting with b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503 not found: ID does not exist" containerID="b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503"
Sep 11 16:27:14.770110 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.770028    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503"} err="failed to get container status \"b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503\": rpc error: code = NotFound desc = could not find container \"b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503\": container with ID starting with b3c330f4c22568d6984c855e47116ac5b93d7138ba1d4dd7c954770845827503 not found: ID does not exist"
Sep 11 16:27:14.770110 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.770051    2559 scope.go:117] "RemoveContainer" containerID="caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1"
Sep 11 16:27:14.778362 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.778343    2559 scope.go:117] "RemoveContainer" containerID="caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1"
Sep 11 16:27:14.778641 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:27:14.778618    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1\": container with ID starting with caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1 not found: ID does not exist" containerID="caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1"
Sep 11 16:27:14.778704 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.778653    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1"} err="failed to get container status \"caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1\": rpc error: code = NotFound desc = could not find container \"caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1\": container with ID starting with caafab584d8c910c7d4931a099904a31ce4f9ee90fcbeb3759065d7af3771ba1 not found: ID does not exist"
Sep 11 16:27:14.791571 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.791539    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"]
Sep 11 16:27:14.797665 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.797639    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-59f108-f7dd4bfd9-7jnhw"]
Sep 11 16:27:14.814852 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.814819    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"]
Sep 11 16:27:14.820312 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:14.820283    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59f108-64f8f7856b-bx9sp"]
Sep 11 16:27:16.034118 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:16.034070    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56dff037-59a9-44d9-ad0e-fa2c61e3df8b" path="/var/lib/kubelet/pods/56dff037-59a9-44d9-ad0e-fa2c61e3df8b/volumes"
Sep 11 16:27:16.034682 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:16.034659    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82dff3fe-b23b-4ca4-9552-45f9ca40a930" path="/var/lib/kubelet/pods/82dff3fe-b23b-4ca4-9552-45f9ca40a930/volumes"
Sep 11 16:27:21.520817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.520780    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"]
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521141    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925b38b3-f013-4623-8050-2090b71ac95b" containerName="revoke-keys"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521152    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="925b38b3-f013-4623-8050-2090b71ac95b" containerName="revoke-keys"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521172    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82dff3fe-b23b-4ca4-9552-45f9ca40a930" containerName="payload-processing"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521178    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82dff3fe-b23b-4ca4-9552-45f9ca40a930" containerName="payload-processing"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521189    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56dff037-59a9-44d9-ad0e-fa2c61e3df8b" containerName="payload-pre-processing"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521195    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="56dff037-59a9-44d9-ad0e-fa2c61e3df8b" containerName="payload-pre-processing"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521269    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82dff3fe-b23b-4ca4-9552-45f9ca40a930" containerName="payload-processing"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521279    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="925b38b3-f013-4623-8050-2090b71ac95b" containerName="revoke-keys"
Sep 11 16:27:21.521310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.521290    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="56dff037-59a9-44d9-ad0e-fa2c61e3df8b" containerName="payload-pre-processing"
Sep 11 16:27:21.526000 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.525980    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"
Sep 11 16:27:21.528721 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.528698    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:27:21.533222 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.533192    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"]
Sep 11 16:27:21.642077 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.642033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b3a64-081a-43c5-acff-6d560056c8e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg\" (UID: \"675b3a64-081a-43c5-acff-6d560056c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"
Sep 11 16:27:21.742815 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.742780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b3a64-081a-43c5-acff-6d560056c8e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg\" (UID: \"675b3a64-081a-43c5-acff-6d560056c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"
Sep 11 16:27:21.743632 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.743596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b3a64-081a-43c5-acff-6d560056c8e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg\" (UID: \"675b3a64-081a-43c5-acff-6d560056c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"
Sep 11 16:27:21.838175 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:21.838074    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"
Sep 11 16:27:22.116151 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:22.116120    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"]
Sep 11 16:27:22.118136 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:27:22.118107    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b3a64_081a_43c5_acff_6d560056c8e4.slice/crio-fbd5ffe11ce09b3f7e6f140ae52ded8ae5680ebe3b2c3d88b2f9b94705323c2b WatchSource:0}: Error finding container fbd5ffe11ce09b3f7e6f140ae52ded8ae5680ebe3b2c3d88b2f9b94705323c2b: Status 404 returned error can't find the container with id fbd5ffe11ce09b3f7e6f140ae52ded8ae5680ebe3b2c3d88b2f9b94705323c2b
Sep 11 16:27:22.805712 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:22.805675    2559 generic.go:356] "Generic (PLEG): container finished" podID="675b3a64-081a-43c5-acff-6d560056c8e4" containerID="9ed4b55ae01f025651763ccc9b34e660c9ba5f48e36710ede7d1efb22589ddf8" exitCode=0
Sep 11 16:27:22.806076 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:22.805714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg" event={"ID":"675b3a64-081a-43c5-acff-6d560056c8e4","Type":"ContainerDied","Data":"9ed4b55ae01f025651763ccc9b34e660c9ba5f48e36710ede7d1efb22589ddf8"}
Sep 11 16:27:22.806076 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:22.805779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg" event={"ID":"675b3a64-081a-43c5-acff-6d560056c8e4","Type":"ContainerStarted","Data":"fbd5ffe11ce09b3f7e6f140ae52ded8ae5680ebe3b2c3d88b2f9b94705323c2b"}
Sep 11 16:27:23.945405 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:23.945379    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"
Sep 11 16:27:24.066966 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:24.066860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b3a64-081a-43c5-acff-6d560056c8e4-maas-api-service-ca\") pod \"675b3a64-081a-43c5-acff-6d560056c8e4\" (UID: \"675b3a64-081a-43c5-acff-6d560056c8e4\") "
Sep 11 16:27:24.067338 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:24.067307    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675b3a64-081a-43c5-acff-6d560056c8e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "675b3a64-081a-43c5-acff-6d560056c8e4" (UID: "675b3a64-081a-43c5-acff-6d560056c8e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:24.168315 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:24.168227    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b3a64-081a-43c5-acff-6d560056c8e4-maas-api-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:24.816162 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:24.816115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg" event={"ID":"675b3a64-081a-43c5-acff-6d560056c8e4","Type":"ContainerDied","Data":"fbd5ffe11ce09b3f7e6f140ae52ded8ae5680ebe3b2c3d88b2f9b94705323c2b"}
Sep 11 16:27:24.816366 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:24.816174    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbd5ffe11ce09b3f7e6f140ae52ded8ae5680ebe3b2c3d88b2f9b94705323c2b"
Sep 11 16:27:24.816366 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:24.816130    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"
Sep 11 16:27:32.122817 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.122781    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"]
Sep 11 16:27:32.123356 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.123062    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh" podUID="a53a2eba-bdd7-44ad-8839-3632ca23166f" containerName="payload-processing" containerID="cri-o://1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb" gracePeriod=30
Sep 11 16:27:32.320664 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.320628    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"]
Sep 11 16:27:32.323078 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.323039    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1e0e69-wj9cg"]
Sep 11 16:27:32.367846 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.367815    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:27:32.447283 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.447177    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g6fc\" (UniqueName: \"kubernetes.io/projected/a53a2eba-bdd7-44ad-8839-3632ca23166f-kube-api-access-2g6fc\") pod \"a53a2eba-bdd7-44ad-8839-3632ca23166f\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") "
Sep 11 16:27:32.447437 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.447302    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53a2eba-bdd7-44ad-8839-3632ca23166f-plugins-config-volume\") pod \"a53a2eba-bdd7-44ad-8839-3632ca23166f\" (UID: \"a53a2eba-bdd7-44ad-8839-3632ca23166f\") "
Sep 11 16:27:32.447690 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.447666    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53a2eba-bdd7-44ad-8839-3632ca23166f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a53a2eba-bdd7-44ad-8839-3632ca23166f" (UID: "a53a2eba-bdd7-44ad-8839-3632ca23166f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:32.449296 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.449268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53a2eba-bdd7-44ad-8839-3632ca23166f-kube-api-access-2g6fc" (OuterVolumeSpecName: "kube-api-access-2g6fc") pod "a53a2eba-bdd7-44ad-8839-3632ca23166f" (UID: "a53a2eba-bdd7-44ad-8839-3632ca23166f"). InnerVolumeSpecName "kube-api-access-2g6fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:32.548455 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.548413    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g6fc\" (UniqueName: \"kubernetes.io/projected/a53a2eba-bdd7-44ad-8839-3632ca23166f-kube-api-access-2g6fc\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.548455 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.548446    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53a2eba-bdd7-44ad-8839-3632ca23166f-plugins-config-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.860725 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.860691    2559 generic.go:356] "Generic (PLEG): container finished" podID="a53a2eba-bdd7-44ad-8839-3632ca23166f" containerID="1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb" exitCode=0
Sep 11 16:27:32.860919 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.860738    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"
Sep 11 16:27:32.860919 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.860777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh" event={"ID":"a53a2eba-bdd7-44ad-8839-3632ca23166f","Type":"ContainerDied","Data":"1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb"}
Sep 11 16:27:32.860919 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.860832    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh" event={"ID":"a53a2eba-bdd7-44ad-8839-3632ca23166f","Type":"ContainerDied","Data":"c1906ae5b41e23f62a19a73bfe683812ec390a66f42842c9853c2f25007d8b80"}
Sep 11 16:27:32.860919 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.860856    2559 scope.go:117] "RemoveContainer" containerID="1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb"
Sep 11 16:27:32.869937 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.869915    2559 scope.go:117] "RemoveContainer" containerID="1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb"
Sep 11 16:27:32.870249 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:27:32.870213    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb\": container with ID starting with 1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb not found: ID does not exist" containerID="1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb"
Sep 11 16:27:32.870335 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.870262    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb"} err="failed to get container status \"1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb\": rpc error: code = NotFound desc = could not find container \"1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb\": container with ID starting with 1b180982ae0bd99f647340f25c8e17df47ba01240653fd999de3839b0ffe0dfb not found: ID does not exist"
Sep 11 16:27:32.883419 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.883390    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"]
Sep 11 16:27:32.888081 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:32.888053    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1e0e69-7cf8966d49-fdmmh"]
Sep 11 16:27:34.033689 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:34.033657    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675b3a64-081a-43c5-acff-6d560056c8e4" path="/var/lib/kubelet/pods/675b3a64-081a-43c5-acff-6d560056c8e4/volumes"
Sep 11 16:27:34.034094 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:27:34.034078    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53a2eba-bdd7-44ad-8839-3632ca23166f" path="/var/lib/kubelet/pods/a53a2eba-bdd7-44ad-8839-3632ca23166f/volumes"
Sep 11 16:28:04.390898 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:28:04.390808    2559 scope.go:117] "RemoveContainer" containerID="a69fe3f8038e32d9fca4f48c917e0dcbf2e5c5b2ccf9da953eb000f4d8352597"
Sep 11 16:28:04.403789 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:28:04.403765    2559 scope.go:117] "RemoveContainer" containerID="c508251050cc223bf61440aaeb77b5408350a89c20abc346d584caf4d798a0bc"
Sep 11 16:30:00.133787 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.133749    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms"]
Sep 11 16:30:00.134227 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.134125    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53a2eba-bdd7-44ad-8839-3632ca23166f" containerName="payload-processing"
Sep 11 16:30:00.134227 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.134138    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53a2eba-bdd7-44ad-8839-3632ca23166f" containerName="payload-processing"
Sep 11 16:30:00.134227 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.134175    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675b3a64-081a-43c5-acff-6d560056c8e4" containerName="revoke-keys"
Sep 11 16:30:00.134227 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.134181    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="675b3a64-081a-43c5-acff-6d560056c8e4" containerName="revoke-keys"
Sep 11 16:30:00.134227 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.134252    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53a2eba-bdd7-44ad-8839-3632ca23166f" containerName="payload-processing"
Sep 11 16:30:00.134436 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.134263    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="675b3a64-081a-43c5-acff-6d560056c8e4" containerName="revoke-keys"
Sep 11 16:30:00.137443 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.137425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms"
Sep 11 16:30:00.140676 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.140650    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\""
Sep 11 16:30:00.147031 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.147007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms"]
Sep 11 16:30:00.147530 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.147511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms"
Sep 11 16:30:00.421288 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.421259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms"]
Sep 11 16:30:00.423126 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:30:00.423092    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6370e865_cf6f_4034_bf59_72b8137cee79.slice/crio-8076a840530af553c2e92b2119ce44718fef8ad2c3d1157a3b7556e3e7d3dfc3 WatchSource:0}: Error finding container 8076a840530af553c2e92b2119ce44718fef8ad2c3d1157a3b7556e3e7d3dfc3: Status 404 returned error can't find the container with id 8076a840530af553c2e92b2119ce44718fef8ad2c3d1157a3b7556e3e7d3dfc3
Sep 11 16:30:00.562511 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:00.562471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms" event={"ID":"6370e865-cf6f-4034-bf59-72b8137cee79","Type":"ContainerStarted","Data":"8076a840530af553c2e92b2119ce44718fef8ad2c3d1157a3b7556e3e7d3dfc3"}
Sep 11 16:30:01.569127 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:01.569091    2559 generic.go:356] "Generic (PLEG): container finished" podID="6370e865-cf6f-4034-bf59-72b8137cee79" containerID="6f75eccc235b7e4192c94579817355cca414b73581391ecc0a007a1523a7038c" exitCode=0
Sep 11 16:30:01.569560 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:01.569153    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms" event={"ID":"6370e865-cf6f-4034-bf59-72b8137cee79","Type":"ContainerDied","Data":"6f75eccc235b7e4192c94579817355cca414b73581391ecc0a007a1523a7038c"}
Sep 11 16:30:02.703920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:02.703893    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms"
Sep 11 16:30:03.579730 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:03.579702    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms"
Sep 11 16:30:03.579730 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:03.579705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819070-7z4ms" event={"ID":"6370e865-cf6f-4034-bf59-72b8137cee79","Type":"ContainerDied","Data":"8076a840530af553c2e92b2119ce44718fef8ad2c3d1157a3b7556e3e7d3dfc3"}
Sep 11 16:30:03.579942 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:30:03.579757    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8076a840530af553c2e92b2119ce44718fef8ad2c3d1157a3b7556e3e7d3dfc3"
Sep 11 16:31:55.829654 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.829618    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"]
Sep 11 16:31:55.830108 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.830010    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6370e865-cf6f-4034-bf59-72b8137cee79" containerName="cleanup"
Sep 11 16:31:55.830108 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.830023    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6370e865-cf6f-4034-bf59-72b8137cee79" containerName="cleanup"
Sep 11 16:31:55.830108 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.830079    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6370e865-cf6f-4034-bf59-72b8137cee79" containerName="cleanup"
Sep 11 16:31:55.832748 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.832730    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:55.836658 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.836639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2f2eeae5-kserve-self-signed-certs\""
Sep 11 16:31:55.858591 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.858560    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"]
Sep 11 16:31:55.965296 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.965224    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-tmp-dir\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:55.965479 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.965327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-dshm\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:55.965479 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.965365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25de4a6e-1c3c-4edc-a371-70b58951922f-tls-certs\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:55.965479 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.965391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-home\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:55.965479 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.965414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szfv\" (UniqueName: \"kubernetes.io/projected/25de4a6e-1c3c-4edc-a371-70b58951922f-kube-api-access-2szfv\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:55.965625 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:55.965525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-model-cache\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.066678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.066627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-tmp-dir\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.066678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.066676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-dshm\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.066938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.066693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25de4a6e-1c3c-4edc-a371-70b58951922f-tls-certs\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.066938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.066710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-home\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.066938 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.066846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szfv\" (UniqueName: \"kubernetes.io/projected/25de4a6e-1c3c-4edc-a371-70b58951922f-kube-api-access-2szfv\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.067093 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.067015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-model-cache\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.067093 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.067084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-home\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.067178 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.067131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-tmp-dir\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.067359 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.067334    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-model-cache\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.069205 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.069171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-dshm\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.069343 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.069313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25de4a6e-1c3c-4edc-a371-70b58951922f-tls-certs\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.078080 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.078060    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szfv\" (UniqueName: \"kubernetes.io/projected/25de4a6e-1c3c-4edc-a371-70b58951922f-kube-api-access-2szfv\") pod \"e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") " pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.141483 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.141391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:31:56.416337 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:56.416310    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"]
Sep 11 16:31:56.418439 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:31:56.418416    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25de4a6e_1c3c_4edc_a371_70b58951922f.slice/crio-7aca89c78ba7af89b800c618408f99d42b95ac5a571cb885cd2643182deefcaf WatchSource:0}: Error finding container 7aca89c78ba7af89b800c618408f99d42b95ac5a571cb885cd2643182deefcaf: Status 404 returned error can't find the container with id 7aca89c78ba7af89b800c618408f99d42b95ac5a571cb885cd2643182deefcaf
Sep 11 16:31:57.110951 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:57.110918    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw" event={"ID":"25de4a6e-1c3c-4edc-a371-70b58951922f","Type":"ContainerStarted","Data":"ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463"}
Sep 11 16:31:57.110951 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:57.110959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw" event={"ID":"25de4a6e-1c3c-4edc-a371-70b58951922f","Type":"ContainerStarted","Data":"7aca89c78ba7af89b800c618408f99d42b95ac5a571cb885cd2643182deefcaf"}
Sep 11 16:31:57.134310 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:31:57.134246    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw" podStartSLOduration=2.134214159 podStartE2EDuration="2.134214159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:57.131779115 +0000 UTC m=+2213.555596811" watchObservedRunningTime="2026-09-11 16:31:57.134214159 +0000 UTC m=+2213.558031852"
Sep 11 16:32:05.252432 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.252388    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"]
Sep 11 16:32:05.255994 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.255973    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.262727 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.262700    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2f2eeae5-kserve-self-signed-certs\""
Sep 11 16:32:05.303876 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.303838    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"]
Sep 11 16:32:05.354532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.354501    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tmp-dir\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.354725 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.354549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-home\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.354725 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.354646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-model-cache\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.354725 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.354692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrnn\" (UniqueName: \"kubernetes.io/projected/e1a2686f-79b3-4c41-b1d0-f330583cc85e-kube-api-access-hcrnn\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.354725 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.354712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tls-certs\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.354914 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.354779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-dshm\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456188 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-home\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456419 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-model-cache\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456419 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456347    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrnn\" (UniqueName: \"kubernetes.io/projected/e1a2686f-79b3-4c41-b1d0-f330583cc85e-kube-api-access-hcrnn\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456419 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tls-certs\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456600 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-dshm\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456588    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tmp-dir\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-home\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456779 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456688    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-model-cache\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.456941 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.456920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tmp-dir\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.458675 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.458656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-dshm\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.458953 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.458933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tls-certs\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.474745 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.474715    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrnn\" (UniqueName: \"kubernetes.io/projected/e1a2686f-79b3-4c41-b1d0-f330583cc85e-kube-api-access-hcrnn\") pod \"e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") " pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.565483 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.565374    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:05.861920 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:05.861840    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"]
Sep 11 16:32:05.864993 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:32:05.864965    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1a2686f_79b3_4c41_b1d0_f330583cc85e.slice/crio-07fa967f65573f7b7c24211cde066d915c3c653e5675dec046c42981dc1de7db WatchSource:0}: Error finding container 07fa967f65573f7b7c24211cde066d915c3c653e5675dec046c42981dc1de7db: Status 404 returned error can't find the container with id 07fa967f65573f7b7c24211cde066d915c3c653e5675dec046c42981dc1de7db
Sep 11 16:32:06.141906 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:06.141804    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:32:06.141906 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:06.141872    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:32:06.154265 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:06.154215    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:32:06.160074 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:06.160038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6" event={"ID":"e1a2686f-79b3-4c41-b1d0-f330583cc85e","Type":"ContainerStarted","Data":"55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b"}
Sep 11 16:32:06.160267 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:06.160080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6" event={"ID":"e1a2686f-79b3-4c41-b1d0-f330583cc85e","Type":"ContainerStarted","Data":"07fa967f65573f7b7c24211cde066d915c3c653e5675dec046c42981dc1de7db"}
Sep 11 16:32:06.171376 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:06.171346    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:32:06.228792 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:06.228737    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6" podStartSLOduration=1.22871782 podStartE2EDuration="1.22871782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:06.226935592 +0000 UTC m=+2222.650753302" watchObservedRunningTime="2026-09-11 16:32:06.22871782 +0000 UTC m=+2222.652535553"
Sep 11 16:32:15.565923 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:15.565881    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:15.565923 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:15.565935    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:15.578862 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:15.578831    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:16.223037 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:16.223004    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:24.562818 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:24.562784    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"]
Sep 11 16:32:26.353631 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:26.353535    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"]
Sep 11 16:32:39.581702 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.581637    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw" podUID="25de4a6e-1c3c-4edc-a371-70b58951922f" containerName="main" containerID="cri-o://ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463" gracePeriod=45
Sep 11 16:32:39.822738 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.822708    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:32:39.974758 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.974658    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25de4a6e-1c3c-4edc-a371-70b58951922f-tls-certs\") pod \"25de4a6e-1c3c-4edc-a371-70b58951922f\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") "
Sep 11 16:32:39.974758 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.974731    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2szfv\" (UniqueName: \"kubernetes.io/projected/25de4a6e-1c3c-4edc-a371-70b58951922f-kube-api-access-2szfv\") pod \"25de4a6e-1c3c-4edc-a371-70b58951922f\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") "
Sep 11 16:32:39.974758 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.974759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-dshm\") pod \"25de4a6e-1c3c-4edc-a371-70b58951922f\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") "
Sep 11 16:32:39.975072 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.974784    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-tmp-dir\") pod \"25de4a6e-1c3c-4edc-a371-70b58951922f\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") "
Sep 11 16:32:39.975072 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.974840    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-model-cache\") pod \"25de4a6e-1c3c-4edc-a371-70b58951922f\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") "
Sep 11 16:32:39.975072 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.974862    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-home\") pod \"25de4a6e-1c3c-4edc-a371-70b58951922f\" (UID: \"25de4a6e-1c3c-4edc-a371-70b58951922f\") "
Sep 11 16:32:39.975218 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.975143    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25de4a6e-1c3c-4edc-a371-70b58951922f" (UID: "25de4a6e-1c3c-4edc-a371-70b58951922f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:39.975218 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.975190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-model-cache" (OuterVolumeSpecName: "model-cache") pod "25de4a6e-1c3c-4edc-a371-70b58951922f" (UID: "25de4a6e-1c3c-4edc-a371-70b58951922f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:39.975218 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.975203    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-home" (OuterVolumeSpecName: "home") pod "25de4a6e-1c3c-4edc-a371-70b58951922f" (UID: "25de4a6e-1c3c-4edc-a371-70b58951922f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:39.977046 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.977019    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25de4a6e-1c3c-4edc-a371-70b58951922f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25de4a6e-1c3c-4edc-a371-70b58951922f" (UID: "25de4a6e-1c3c-4edc-a371-70b58951922f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:32:39.977046 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.977031    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-dshm" (OuterVolumeSpecName: "dshm") pod "25de4a6e-1c3c-4edc-a371-70b58951922f" (UID: "25de4a6e-1c3c-4edc-a371-70b58951922f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:39.977204 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:39.977064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25de4a6e-1c3c-4edc-a371-70b58951922f-kube-api-access-2szfv" (OuterVolumeSpecName: "kube-api-access-2szfv") pod "25de4a6e-1c3c-4edc-a371-70b58951922f" (UID: "25de4a6e-1c3c-4edc-a371-70b58951922f"). InnerVolumeSpecName "kube-api-access-2szfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:32:40.075843 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.075806    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2szfv\" (UniqueName: \"kubernetes.io/projected/25de4a6e-1c3c-4edc-a371-70b58951922f-kube-api-access-2szfv\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:40.075843 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.075838    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:40.075843 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.075847    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:40.076067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.075858    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:40.076067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.075867    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25de4a6e-1c3c-4edc-a371-70b58951922f-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:40.076067 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.075876    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25de4a6e-1c3c-4edc-a371-70b58951922f-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:40.331324 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.331281    2559 generic.go:356] "Generic (PLEG): container finished" podID="25de4a6e-1c3c-4edc-a371-70b58951922f" containerID="ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463" exitCode=0
Sep 11 16:32:40.331532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.331331    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"
Sep 11 16:32:40.331532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.331330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw" event={"ID":"25de4a6e-1c3c-4edc-a371-70b58951922f","Type":"ContainerDied","Data":"ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463"}
Sep 11 16:32:40.331532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.331387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw" event={"ID":"25de4a6e-1c3c-4edc-a371-70b58951922f","Type":"ContainerDied","Data":"7aca89c78ba7af89b800c618408f99d42b95ac5a571cb885cd2643182deefcaf"}
Sep 11 16:32:40.331532 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.331410    2559 scope.go:117] "RemoveContainer" containerID="ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463"
Sep 11 16:32:40.340379 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.340355    2559 scope.go:117] "RemoveContainer" containerID="ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463"
Sep 11 16:32:40.340657 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:32:40.340638    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463\": container with ID starting with ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463 not found: ID does not exist" containerID="ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463"
Sep 11 16:32:40.340733 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.340673    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463"} err="failed to get container status \"ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463\": rpc error: code = NotFound desc = could not find container \"ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463\": container with ID starting with ac9743da58e612948317edab72b0e0fdc49bbdb0ca62203acb882bc0f9e9b463 not found: ID does not exist"
Sep 11 16:32:40.351976 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.351947    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"]
Sep 11 16:32:40.358373 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:40.358336    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2f2eeae5-kserve-65d87896c-w2pmw"]
Sep 11 16:32:41.372634 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.372596    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6" podUID="e1a2686f-79b3-4c41-b1d0-f330583cc85e" containerName="main" containerID="cri-o://55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b" gracePeriod=45
Sep 11 16:32:41.633260 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.633175    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:41.792390 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792361    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tls-certs\") pod \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") "
Sep 11 16:32:41.792569 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792423    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-dshm\") pod \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") "
Sep 11 16:32:41.792569 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792513    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-model-cache\") pod \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") "
Sep 11 16:32:41.792569 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792551    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcrnn\" (UniqueName: \"kubernetes.io/projected/e1a2686f-79b3-4c41-b1d0-f330583cc85e-kube-api-access-hcrnn\") pod \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") "
Sep 11 16:32:41.792569 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792571    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-home\") pod \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") "
Sep 11 16:32:41.792821 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792599    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tmp-dir\") pod \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\" (UID: \"e1a2686f-79b3-4c41-b1d0-f330583cc85e\") "
Sep 11 16:32:41.792821 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792779    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1a2686f-79b3-4c41-b1d0-f330583cc85e" (UID: "e1a2686f-79b3-4c41-b1d0-f330583cc85e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:41.792925 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792859    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-home" (OuterVolumeSpecName: "home") pod "e1a2686f-79b3-4c41-b1d0-f330583cc85e" (UID: "e1a2686f-79b3-4c41-b1d0-f330583cc85e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:41.792983 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792929    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1a2686f-79b3-4c41-b1d0-f330583cc85e" (UID: "e1a2686f-79b3-4c41-b1d0-f330583cc85e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:41.793046 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.792993    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:41.793046 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.793008    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:41.793046 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.793017    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:41.794521 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.794492    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1a2686f-79b3-4c41-b1d0-f330583cc85e" (UID: "e1a2686f-79b3-4c41-b1d0-f330583cc85e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:32:41.794521 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.794507    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-dshm" (OuterVolumeSpecName: "dshm") pod "e1a2686f-79b3-4c41-b1d0-f330583cc85e" (UID: "e1a2686f-79b3-4c41-b1d0-f330583cc85e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:41.794670 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.794627    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a2686f-79b3-4c41-b1d0-f330583cc85e-kube-api-access-hcrnn" (OuterVolumeSpecName: "kube-api-access-hcrnn") pod "e1a2686f-79b3-4c41-b1d0-f330583cc85e" (UID: "e1a2686f-79b3-4c41-b1d0-f330583cc85e"). InnerVolumeSpecName "kube-api-access-hcrnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:32:41.894533 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.894429    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a2686f-79b3-4c41-b1d0-f330583cc85e-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:41.894533 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.894467    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a2686f-79b3-4c41-b1d0-f330583cc85e-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:41.894533 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:41.894480    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcrnn\" (UniqueName: \"kubernetes.io/projected/e1a2686f-79b3-4c41-b1d0-f330583cc85e-kube-api-access-hcrnn\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\""
Sep 11 16:32:42.033801 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.033764    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25de4a6e-1c3c-4edc-a371-70b58951922f" path="/var/lib/kubelet/pods/25de4a6e-1c3c-4edc-a371-70b58951922f/volumes"
Sep 11 16:32:42.343537 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.343496    2559 generic.go:356] "Generic (PLEG): container finished" podID="e1a2686f-79b3-4c41-b1d0-f330583cc85e" containerID="55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b" exitCode=0
Sep 11 16:32:42.343759 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.343557    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"
Sep 11 16:32:42.343759 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.343578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6" event={"ID":"e1a2686f-79b3-4c41-b1d0-f330583cc85e","Type":"ContainerDied","Data":"55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b"}
Sep 11 16:32:42.343759 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.343622    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6" event={"ID":"e1a2686f-79b3-4c41-b1d0-f330583cc85e","Type":"ContainerDied","Data":"07fa967f65573f7b7c24211cde066d915c3c653e5675dec046c42981dc1de7db"}
Sep 11 16:32:42.343759 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.343640    2559 scope.go:117] "RemoveContainer" containerID="55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b"
Sep 11 16:32:42.352798 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.352781    2559 scope.go:117] "RemoveContainer" containerID="55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b"
Sep 11 16:32:42.353080 ip-10-0-129-137 kubenswrapper[2559]: E0911 16:32:42.353060    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b\": container with ID starting with 55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b not found: ID does not exist" containerID="55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b"
Sep 11 16:32:42.353119 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.353089    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b"} err="failed to get container status \"55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b\": rpc error: code = NotFound desc = could not find container \"55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b\": container with ID starting with 55345273385ca742a475138555153a806fd3d7725ea005c148475d3128ca163b not found: ID does not exist"
Sep 11 16:32:42.362648 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.362622    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"]
Sep 11 16:32:42.366596 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:42.366568    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2f2eeae5-kserve-bc6db7855-vx8d6"]
Sep 11 16:32:44.033678 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:44.033645    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1a2686f-79b3-4c41-b1d0-f330583cc85e" path="/var/lib/kubelet/pods/e1a2686f-79b3-4c41-b1d0-f330583cc85e/volumes"
Sep 11 16:32:51.106968 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.106928    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"]
Sep 11 16:32:51.107605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.107488    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a2686f-79b3-4c41-b1d0-f330583cc85e" containerName="main"
Sep 11 16:32:51.107605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.107510    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a2686f-79b3-4c41-b1d0-f330583cc85e" containerName="main"
Sep 11 16:32:51.107605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.107552    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25de4a6e-1c3c-4edc-a371-70b58951922f" containerName="main"
Sep 11 16:32:51.107605 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.107560    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25de4a6e-1c3c-4edc-a371-70b58951922f" containerName="main"
Sep 11 16:32:51.107832 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.107641    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="25de4a6e-1c3c-4edc-a371-70b58951922f" containerName="main"
Sep 11 16:32:51.107832 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.107655    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1a2686f-79b3-4c41-b1d0-f330583cc85e" containerName="main"
Sep 11 16:32:51.111973 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.111952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.116012 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.115989    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 16:32:51.117086 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.117060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfbz8\""
Sep 11 16:32:51.117462 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.117442    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 16:32:51.128633 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.128607    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"]
Sep 11 16:32:51.281520 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.281487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c645d2b-b222-458d-b57f-24fa828c95b4-metrics-certs\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.281695 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.281607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c645d2b-b222-458d-b57f-24fa828c95b4-openshift-service-ca\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.281695 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.281640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfft8\" (UniqueName: \"kubernetes.io/projected/6c645d2b-b222-458d-b57f-24fa828c95b4-kube-api-access-dfft8\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.281695 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.281690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c645d2b-b222-458d-b57f-24fa828c95b4-maas-api-tls\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.383060 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.382964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c645d2b-b222-458d-b57f-24fa828c95b4-maas-api-tls\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.383060 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.383020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c645d2b-b222-458d-b57f-24fa828c95b4-metrics-certs\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.383398 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.383081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c645d2b-b222-458d-b57f-24fa828c95b4-openshift-service-ca\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.383398 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.383099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfft8\" (UniqueName: \"kubernetes.io/projected/6c645d2b-b222-458d-b57f-24fa828c95b4-kube-api-access-dfft8\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.383808 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.383785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c645d2b-b222-458d-b57f-24fa828c95b4-openshift-service-ca\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.385648 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.385614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c645d2b-b222-458d-b57f-24fa828c95b4-maas-api-tls\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.385648 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.385630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c645d2b-b222-458d-b57f-24fa828c95b4-metrics-certs\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.393888 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.393865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfft8\" (UniqueName: \"kubernetes.io/projected/6c645d2b-b222-458d-b57f-24fa828c95b4-kube-api-access-dfft8\") pod \"maas-api-984bf6b7d-7csd8\" (UID: \"6c645d2b-b222-458d-b57f-24fa828c95b4\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.422942 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.422895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:51.697740 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.697712    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"]
Sep 11 16:32:51.699914 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:32:51.699884    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c645d2b_b222_458d_b57f_24fa828c95b4.slice/crio-6c554817c07a9a7ae0ed3cc3dc1802c6a2b73e34e92e11f6931fb22381c00b8f WatchSource:0}: Error finding container 6c554817c07a9a7ae0ed3cc3dc1802c6a2b73e34e92e11f6931fb22381c00b8f: Status 404 returned error can't find the container with id 6c554817c07a9a7ae0ed3cc3dc1802c6a2b73e34e92e11f6931fb22381c00b8f
Sep 11 16:32:51.701300 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:51.701283    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:32:52.396903 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:52.396862    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8" event={"ID":"6c645d2b-b222-458d-b57f-24fa828c95b4","Type":"ContainerStarted","Data":"6c554817c07a9a7ae0ed3cc3dc1802c6a2b73e34e92e11f6931fb22381c00b8f"}
Sep 11 16:32:54.409975 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:54.409937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8" event={"ID":"6c645d2b-b222-458d-b57f-24fa828c95b4","Type":"ContainerStarted","Data":"c74550907db7993a5463c5a410e54e6e876c1309d2e7caa56acefa901df5f26e"}
Sep 11 16:32:54.410391 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:54.410060    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:32:54.429608 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:32:54.429550    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8" podStartSLOduration=1.159527625 podStartE2EDuration="3.429534502s" totalImagesPullingTime="2.270006877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:54.427789406 +0000 UTC m=+2270.851607117" watchObservedRunningTime="2026-09-11 16:32:54.429534502 +0000 UTC m=+2270.853352198"
Sep 11 16:33:00.419881 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:00.419848    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-7csd8"
Sep 11 16:33:04.465863 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:04.465827    2559 scope.go:117] "RemoveContainer" containerID="76bce976d1d21b29d45a53d6e87bc62d48c9ed98a9d3a6c2fa3056d968dfb7ea"
Sep 11 16:33:04.474391 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:04.474367    2559 scope.go:117] "RemoveContainer" containerID="34561a71a2e92379957cc6821c7dfbd555a3322b8157e1094dc0b8e88ca5f962"
Sep 11 16:33:19.102528 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:19.102496    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-jq8r7_dfc98944-c2b3-4087-bd88-f1f091c02f69/copy-manifests/0.log"
Sep 11 16:33:19.109226 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:19.109194    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-jq8r7_dfc98944-c2b3-4087-bd88-f1f091c02f69/manager/0.log"
Sep 11 16:33:19.714758 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:19.714726    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-984bf6b7d-7csd8_6c645d2b-b222-458d-b57f-24fa828c95b4/maas-api/0.log"
Sep 11 16:33:20.054686 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:20.054645    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819070-7z4ms_6370e865-cf6f-4034-bf59-72b8137cee79/cleanup/0.log"
Sep 11 16:33:20.170050 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:20.170019    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-7hprz_fcb06b3c-fdee-43a1-85b0-a9ede25197a0/postgres/0.log"
Sep 11 16:33:21.273708 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:21.273681    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tx82k_a66b9839-f4ca-4b4b-aea6-08012f472837/kuadrant-console-plugin/0.log"
Sep 11 16:33:22.215359 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:22.215326    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-52wr4_c3e2870b-bbbd-4422-bc75-23321a7efa18/kube-auth-proxy/0.log"
Sep 11 16:33:22.571694 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:22.571653    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g5d4s_7f2720e4-65c1-4a80-8b03-5bc595a9a8d2/payload-pre-processing/0.log"
Sep 11 16:33:23.153044 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:23.153013    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-69ksw_f8e68c94-3dbf-4b23-8733-7e8d83cd6a7d/main/0.log"
Sep 11 16:33:23.369628 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:23.369596    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-rc58q_7606f832-d8dd-40ce-9451-50bf9ce12c9c/main/0.log"
Sep 11 16:33:23.592332 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:23.592298    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkxqlt_73f35b9e-d767-420e-98b0-9f2102a05865/main/0.log"
Sep 11 16:33:31.963318 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:31.963286    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nnlqc_c81f8ab6-37ce-477f-9eae-90441a29d9b3/global-pull-secret-syncer/0.log"
Sep 11 16:33:32.129582 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:32.129551    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wl7tc_5a5b87e4-2e32-4545-947d-4482ab103ac5/konnectivity-agent/0.log"
Sep 11 16:33:32.152899 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:32.152863    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-137.ec2.internal_fa926f4c1f4ea217ab8dc7d0d74214a4/haproxy/0.log"
Sep 11 16:33:36.357596 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:36.357565    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tx82k_a66b9839-f4ca-4b4b-aea6-08012f472837/kuadrant-console-plugin/0.log"
Sep 11 16:33:38.611071 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:38.611038    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-rnlnc_ef47f679-d474-400d-83a7-c760fb7fd96f/monitoring-plugin/0.log"
Sep 11 16:33:38.832809 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:38.832778    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g9vvc_5cd34ec5-28af-4e7b-9816-b3aec953d6dc/node-exporter/0.log"
Sep 11 16:33:38.870341 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:38.870310    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g9vvc_5cd34ec5-28af-4e7b-9816-b3aec953d6dc/kube-rbac-proxy/0.log"
Sep 11 16:33:38.902937 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:38.902911    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g9vvc_5cd34ec5-28af-4e7b-9816-b3aec953d6dc/init-textfile/0.log"
Sep 11 16:33:39.056139 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.056103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-q52hv_43dc71f8-7bc8-4ccc-99fd-112a5798b4fc/kube-rbac-proxy-main/0.log"
Sep 11 16:33:39.091886 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.091853    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-q52hv_43dc71f8-7bc8-4ccc-99fd-112a5798b4fc/kube-rbac-proxy-self/0.log"
Sep 11 16:33:39.120596 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.120564    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-q52hv_43dc71f8-7bc8-4ccc-99fd-112a5798b4fc/openshift-state-metrics/0.log"
Sep 11 16:33:39.167928 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.167892    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6dc46a2e-19c8-4f0e-a256-faaeac548d6e/prometheus/0.log"
Sep 11 16:33:39.193526 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.193495    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6dc46a2e-19c8-4f0e-a256-faaeac548d6e/config-reloader/0.log"
Sep 11 16:33:39.224255 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.224213    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6dc46a2e-19c8-4f0e-a256-faaeac548d6e/thanos-sidecar/0.log"
Sep 11 16:33:39.245341 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.245315    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6dc46a2e-19c8-4f0e-a256-faaeac548d6e/kube-rbac-proxy-web/0.log"
Sep 11 16:33:39.278478 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.278451    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6dc46a2e-19c8-4f0e-a256-faaeac548d6e/kube-rbac-proxy/0.log"
Sep 11 16:33:39.310130 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.310054    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6dc46a2e-19c8-4f0e-a256-faaeac548d6e/kube-rbac-proxy-thanos/0.log"
Sep 11 16:33:39.336378 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.336346    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6dc46a2e-19c8-4f0e-a256-faaeac548d6e/init-config-reloader/0.log"
Sep 11 16:33:39.376814 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.376788    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-tpbsc_1f039024-5e0f-44f7-996d-f3fbbf3f3987/prometheus-operator/0.log"
Sep 11 16:33:39.409750 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.409717    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-tpbsc_1f039024-5e0f-44f7-996d-f3fbbf3f3987/kube-rbac-proxy/0.log"
Sep 11 16:33:39.465358 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.465331    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-7fgnj_75986d73-59ba-42ee-81e8-699504af6e0f/prometheus-operator-admission-webhook/0.log"
Sep 11 16:33:39.589201 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.589119    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb98cd666-rflpk_9a13f4b3-6021-4081-8951-3b3f29e2c384/thanos-query/0.log"
Sep 11 16:33:39.621994 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.621930    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb98cd666-rflpk_9a13f4b3-6021-4081-8951-3b3f29e2c384/kube-rbac-proxy-web/0.log"
Sep 11 16:33:39.645144 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.645117    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb98cd666-rflpk_9a13f4b3-6021-4081-8951-3b3f29e2c384/kube-rbac-proxy/0.log"
Sep 11 16:33:39.666731 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.666698    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb98cd666-rflpk_9a13f4b3-6021-4081-8951-3b3f29e2c384/prom-label-proxy/0.log"
Sep 11 16:33:39.690626 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.690598    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb98cd666-rflpk_9a13f4b3-6021-4081-8951-3b3f29e2c384/kube-rbac-proxy-rules/0.log"
Sep 11 16:33:39.713189 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:39.713158    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fb98cd666-rflpk_9a13f4b3-6021-4081-8951-3b3f29e2c384/kube-rbac-proxy-metrics/0.log"
Sep 11 16:33:42.024904 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.024816    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"]
Sep 11 16:33:42.028213 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.028182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.031308 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.031280    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg9tr\"/\"kube-root-ca.crt\""
Sep 11 16:33:42.031454 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.031281    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg9tr\"/\"openshift-service-ca.crt\""
Sep 11 16:33:42.031454 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.031385    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bg9tr\"/\"default-dockercfg-lx7vd\""
Sep 11 16:33:42.039761 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.039734    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"]
Sep 11 16:33:42.058379 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.058339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htkr\" (UniqueName: \"kubernetes.io/projected/9a253ba4-09ff-455e-b8d6-246981cb5b57-kube-api-access-6htkr\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.058558 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.058431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-proc\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.058558 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.058454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-sys\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.058558 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.058484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-podres\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.058558 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.058505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-lib-modules\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.159978 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.159930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-sys\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.159998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-podres\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.160030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-lib-modules\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.160073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-sys\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.160109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htkr\" (UniqueName: \"kubernetes.io/projected/9a253ba4-09ff-455e-b8d6-246981cb5b57-kube-api-access-6htkr\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160199 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.160169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-proc\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160470 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.160223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-podres\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160470 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.160286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-proc\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.160470 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.160289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a253ba4-09ff-455e-b8d6-246981cb5b57-lib-modules\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.169824 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.169796    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htkr\" (UniqueName: \"kubernetes.io/projected/9a253ba4-09ff-455e-b8d6-246981cb5b57-kube-api-access-6htkr\") pod \"perf-node-gather-daemonset-8pk2r\" (UID: \"9a253ba4-09ff-455e-b8d6-246981cb5b57\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.338978 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.338894    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:42.614912 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.614883    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"]
Sep 11 16:33:42.616468 ip-10-0-129-137 kubenswrapper[2559]: W0911 16:33:42.616438    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a253ba4_09ff_455e_b8d6_246981cb5b57.slice/crio-266c55b3c297adb01874f334a344ae2723a2a2453528038aba8dacfcde6fcc2a WatchSource:0}: Error finding container 266c55b3c297adb01874f334a344ae2723a2a2453528038aba8dacfcde6fcc2a: Status 404 returned error can't find the container with id 266c55b3c297adb01874f334a344ae2723a2a2453528038aba8dacfcde6fcc2a
Sep 11 16:33:42.635427 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:42.635399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r" event={"ID":"9a253ba4-09ff-455e-b8d6-246981cb5b57","Type":"ContainerStarted","Data":"266c55b3c297adb01874f334a344ae2723a2a2453528038aba8dacfcde6fcc2a"}
Sep 11 16:33:43.642106 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:43.642069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r" event={"ID":"9a253ba4-09ff-455e-b8d6-246981cb5b57","Type":"ContainerStarted","Data":"9efa9e344a42f5da837927f4a72b49bbbe9e59c72b561c81a1c7f632d7acea49"}
Sep 11 16:33:43.642548 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:43.642177    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:43.661352 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:43.661292    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r" podStartSLOduration=1.661274801 podStartE2EDuration="1.661274801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:43.660450908 +0000 UTC m=+2320.084268613" watchObservedRunningTime="2026-09-11 16:33:43.661274801 +0000 UTC m=+2320.085092494"
Sep 11 16:33:44.485009 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:44.484982    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jtzf7_852587b1-0a81-4241-a4bc-af2730e72a53/dns/0.log"
Sep 11 16:33:44.508372 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:44.508346    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jtzf7_852587b1-0a81-4241-a4bc-af2730e72a53/kube-rbac-proxy/0.log"
Sep 11 16:33:44.584287 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:44.584259    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-78ppt_dabccca7-3201-4e1e-84d7-ef82f064128a/dns-node-resolver/0.log"
Sep 11 16:33:45.173515 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:45.173481    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67579569c8-v62fc_6ab47786-bfd4-4230-983b-fa0d6098b424/registry/0.log"
Sep 11 16:33:45.246921 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:45.246870    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lhrj7_4cab6e56-d6b3-4161-b5ca-34d418afb13f/node-ca/0.log"
Sep 11 16:33:46.816836 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:46.816799    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-52wr4_c3e2870b-bbbd-4422-bc75-23321a7efa18/kube-auth-proxy/0.log"
Sep 11 16:33:46.921604 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:46.921570    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g5d4s_7f2720e4-65c1-4a80-8b03-5bc595a9a8d2/payload-pre-processing/0.log"
Sep 11 16:33:47.938555 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:47.938525    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zl6nf_1fa7ffdf-6c14-451d-88c9-c92eb7ebd9af/serve-healthcheck-canary/0.log"
Sep 11 16:33:48.449379 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:48.449345    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jpsls_59b28b96-2421-41b3-8d69-842d2e47799b/kube-rbac-proxy/0.log"
Sep 11 16:33:48.474425 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:48.474395    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jpsls_59b28b96-2421-41b3-8d69-842d2e47799b/exporter/0.log"
Sep 11 16:33:48.495341 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:48.495308    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jpsls_59b28b96-2421-41b3-8d69-842d2e47799b/extractor/0.log"
Sep 11 16:33:49.657133 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:49.657102    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-8pk2r"
Sep 11 16:33:51.457148 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:51.457114    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-jq8r7_dfc98944-c2b3-4087-bd88-f1f091c02f69/manager/0.log"
Sep 11 16:33:51.481016 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:51.480989    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-jq8r7_dfc98944-c2b3-4087-bd88-f1f091c02f69/copy-manifests/0.log"
Sep 11 16:33:53.020595 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:53.020561    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6848676c-xhm5j_c343d096-a13f-4710-ba42-37b8e9a868f0/manager/0.log"
Sep 11 16:33:53.058337 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:53.058306    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5fllx_49315e01-9b89-47c2-b6c1-d5e6a3cb9477/openshift-lws-operator/0.log"
Sep 11 16:33:53.389184 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:53.389110    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-52wr4_c3e2870b-bbbd-4422-bc75-23321a7efa18/kube-auth-proxy/0.log"
Sep 11 16:33:53.486911 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:53.486879    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g5d4s_7f2720e4-65c1-4a80-8b03-5bc595a9a8d2/payload-pre-processing/0.log"
Sep 11 16:33:55.581124 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:55.581091    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-52wr4_c3e2870b-bbbd-4422-bc75-23321a7efa18/kube-auth-proxy/0.log"
Sep 11 16:33:55.667567 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:55.667527    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-g5d4s_7f2720e4-65c1-4a80-8b03-5bc595a9a8d2/payload-pre-processing/0.log"
Sep 11 16:33:58.335208 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:58.335133    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-chm5p_c6e060cf-52e6-4bd5-a296-51ac19706df0/migrator/0.log"
Sep 11 16:33:58.358634 ip-10-0-129-137 kubenswrapper[2559]: I0911 16:33:58.358599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-chm5p_c6e060cf-52e6-4bd5-a296-51ac19706df0/graceful-termination/0.log"