Sep 10 02:44:53.041660 ip-10-0-142-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 02:44:53.041671 ip-10-0-142-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 02:44:53.041678 ip-10-0-142-156 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 02:44:53.041916 ip-10-0-142-156 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 02:45:02.155315 ip-10-0-142-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 10 02:45:02.155328 ip-10-0-142-156 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 902511a2aa034ba9a689fa661e35f53e --
Sep 10 02:46:41.341321 ip-10-0-142-156 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 02:46:41.787745 ip-10-0-142-156 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 02:46:41.787745 ip-10-0-142-156 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 02:46:41.787745 ip-10-0-142-156 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 02:46:41.787745 ip-10-0-142-156 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 02:46:41.787745 ip-10-0-142-156 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 02:46:41.788847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.788763    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 02:46:41.792736 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792721    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:41.792736 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792737    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792740    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792744    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792748    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792752    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792756    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792759    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792762    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792765    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792768    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792770    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792773    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792784    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792787    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792790    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792793    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792795    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792798    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792800    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792803    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:41.792850 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792806    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792809    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792811    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792814    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792816    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792819    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792821    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792824    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792826    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792829    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792831    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792834    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792836    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792839    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792842    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792846    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792848    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792851    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792853    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792856    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:41.793341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792859    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792862    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792864    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792867    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792869    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792872    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792874    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792877    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792880    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792883    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792885    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792890    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792893    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792896    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792899    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792902    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792905    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792910    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792913    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:41.793823 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792916    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792919    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792922    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792927    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792930    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792933    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792936    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792940    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792943    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792946    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792948    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792952    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792955    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792958    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792960    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792963    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792965    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792968    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792970    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:41.794294 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792973    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792975    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792978    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792980    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792983    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792986    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792988    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792991    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792993    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792996    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.792998    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793001    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793003    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793006    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793009    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793011    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793014    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793016    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793018    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793021    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:41.794778 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793731    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793738    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793742    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793746    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793749    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793753    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793755    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793758    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793760    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793764    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793767    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793770    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793773    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793776    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793778    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793781    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793783    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793786    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793788    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793802    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:41.795285 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793805    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793808    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793811    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793814    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793817    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793820    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793823    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793825    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793828    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793831    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793833    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793836    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793840    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793844    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793847    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793849    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793852    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793855    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793858    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793860    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:41.795779 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793863    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793866    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793868    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793871    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793875    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793878    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793881    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793884    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793887    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793889    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793892    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793894    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793897    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793899    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793902    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793904    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793907    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793910    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793913    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:41.796301 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793915    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793918    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793920    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793923    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793925    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793928    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793931    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793934    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793936    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793939    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793942    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793944    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793947    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793951    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793953    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793956    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793958    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793961    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793963    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793966    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:41.796761 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793968    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793971    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793973    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793976    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793978    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793981    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793983    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793986    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793988    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793991    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793993    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793995    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.793998    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794000    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794003    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794005    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794008    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794014    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794017    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794020    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:41.797264 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794386    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794392    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794397    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794400    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794403    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794406    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794409    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794411    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794414    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794416    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794419    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794421    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794424    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794426    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794429    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794431    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794434    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794437    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794439    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:41.797754 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794442    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794444    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794447    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794450    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794452    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794455    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794458    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794460    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794463    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794465    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794470    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794472    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794475    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794478    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794480    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794483    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794485    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794489    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794492    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794494    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:41.798228 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794497    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794499    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794502    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794504    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794507    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794509    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794512    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794514    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794517    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794519    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794522    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794524    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794527    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794529    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794531    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794536    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794540    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794543    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794547    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:41.798714 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794549    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794552    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794556    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794562    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794565    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794567    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794570    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794573    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794576    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794578    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794581    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794583    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794586    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794589    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794591    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794594    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794597    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794599    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794602    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794604    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:41.799178 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794607    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794609    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794611    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794614    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794617    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794619    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794622    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794624    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794626    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794629    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794631    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794634    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794637    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794639    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794642    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794646    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794648    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794651    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794653    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794656    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:41.799702 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.794659    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794726    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794758    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794763    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794768    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794773    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794777    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794781    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794785    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794788    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794791    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794795    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794798    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794801    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794804    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794807    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794810    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794813    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794816    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794820    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794822    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794825    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794829    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794832    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 02:46:41.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794836    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794839    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794842    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794846    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794850    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794853    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794855    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794859    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794862    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794866    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794868    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794872    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794874    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794877    2580 flags.go:64] FLAG: --enable-server="true"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794880    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794884    2580 flags.go:64] FLAG: --event-burst="100"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794887    2580 flags.go:64] FLAG: --event-qps="50"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794890    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794893    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794896    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794899    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794902    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794905    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794907    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794910    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 02:46:41.800784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794913    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794916    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794918    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794921    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794924    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794927    2580 flags.go:64] FLAG: --feature-gates=""
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794931    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794934    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794936    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794939    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794943    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794946    2580 flags.go:64] FLAG: --help="false"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794949    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-156.ec2.internal"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794952    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794955    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794957    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794961    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794964    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794967    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794970    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794973    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794976    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794979    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794982    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 02:46:41.801419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794985    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794987    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794990    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794993    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794996    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.794999    2580 flags.go:64] FLAG: --lock-file=""
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795001    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795004    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795007    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795012    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795015    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795017    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795020    2580 flags.go:64] FLAG: --logging-format="text"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795023    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795026    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795029    2580 flags.go:64] FLAG: --manifest-url=""
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795031    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795036    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795040    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795044    2580 flags.go:64] FLAG: --max-pods="110"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795047    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795050    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795053    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795056    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795059    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 02:46:41.802008 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795074    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795077    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795084    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795087    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795090    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795093    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795097    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795102    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795105    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795108    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795111    2580 flags.go:64] FLAG: --port="10250"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795114    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795116    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-020a8caad03696390"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795119    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795122    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795125    2580 flags.go:64] FLAG: --register-node="true"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795128    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795132    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795135    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795138    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795140    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795144    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795146    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795149    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795153    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 02:46:41.802638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795176    2580 flags.go:64] FLAG: --runonce="false"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795193    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795199    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795203    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795206    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795209    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795213    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795217    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795220    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795223    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795226    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795229    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795232    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795235    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795238    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795241    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795253    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795256    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795259    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795264    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795267    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795271    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795274    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795277    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795280    2580 flags.go:64] FLAG: --v="2"
Sep 10 02:46:41.803292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795285    2580 flags.go:64] FLAG: --version="false"
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795289    2580 flags.go:64] FLAG: --vmodule=""
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795293    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795297    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795422    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795425    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795428    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795432    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795436    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795439    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795442    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795445    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795448    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795450    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795453    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795455    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795458    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795461    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795463    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795466    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:41.803916 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795468    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795471    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795474    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795476    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795479    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795481    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795484    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795486    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795489    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795492    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795495    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795497    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795500    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795503    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795505    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795508    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795511    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795513    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795516    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795519    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:41.804431 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795521    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795527    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795529    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795532    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795535    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795537    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795540    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795542    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795545    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795547    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795550    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795553    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795557    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795560    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795563    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795566    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795569    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795571    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795574    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:41.804920 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795577    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795579    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795582    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795585    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795587    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795591    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795595    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795598    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795601    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795604    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795607    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795610    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795613    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795616    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795618    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795621    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795624    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795626    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795629    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:41.805420 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795631    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795634    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795637    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795639    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795643    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795647    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795650    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795653    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795655    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795657    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795660    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795663    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795665    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795668    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795670    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795673    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795676    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795679    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795682    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795685    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:41.806025 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795687    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795690    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795695    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795697    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.795700    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.795705    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.803174    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.803189    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.804094    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.804101    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804166    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804171    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804174    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804177    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:41.806782 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804180    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804182    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804185    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804187    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804190    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804193    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804195    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804199    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804203    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804206    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804209    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804212    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804215    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804217    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804220    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804222    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804225    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804227    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804230    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804232    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:41.807197 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804235    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804237    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804240    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804242    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804245    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804248    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804251    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804254    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804257    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804260    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804262    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804265    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804267    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804270    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804272    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804275    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804277    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804280    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804282    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804285    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:41.807700 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804287    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804290    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804292    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804295    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804298    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804301    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804305    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804308    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804311    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804314    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804316    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804319    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804321    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804324    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804327    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804329    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804331    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804334    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804337    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:41.808226 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804340    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804343    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804345    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804348    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804350    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804353    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804355    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804358    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804360    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804363    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804365    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804368    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804370    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804373    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804375    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804378    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804380    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804383    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804385    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804388    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:41.808720 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804391    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804393    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804396    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804398    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804401    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804404    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804406    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804410    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804413    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804416    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804419    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804422    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804424    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804427    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804430    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:41.809489 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804433    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.804438    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804546    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804550    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804553    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804556    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804559    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804562    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804565    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804567    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804569    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804572    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804576    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804580    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:41.809873 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804583    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804586    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804589    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804592    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804595    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804598    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804600    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804603    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804606    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804608    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804611    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804613    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804616    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804618    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804621    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804624    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804627    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804629    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804632    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804635    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:41.810282 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804637    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804639    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804642    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804644    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804647    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804649    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804652    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804654    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804657    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804660    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804662    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804665    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804668    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804671    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804674    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804677    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804680    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804683    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804685    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:41.810759 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804688    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804690    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804693    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804695    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804698    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804700    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804703    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804706    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804708    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804711    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804713    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804716    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804719    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804721    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804724    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804726    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804729    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804731    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804734    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804736    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:41.811254 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804739    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804741    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804743    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804746    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804749    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804751    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804754    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804756    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804758    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804761    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804763    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804766    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804768    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804771    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804773    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804776    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804778    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804781    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804783    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:41.811739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804786    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804789    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804792    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804794    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804796    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804799    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804802    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804804    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:41.804807    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.804811    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.804904    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.809789    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.810778    2580 server.go:1019] "Starting client certificate rotation"
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.810879    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 02:46:41.812225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.810914    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 02:46:41.813632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.813617    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 02:46:41.814708 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.814689    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 02:46:41.826994 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.826977    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 10 02:46:41.830134 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.830119    2580 log.go:25] "Validated CRI v1 image API"
Sep 10 02:46:41.832078 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.832043    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 02:46:41.834565 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.834537    2580 fs.go:135] Filesystem UUIDs: map[6bd25627-647b-45ea-854f-88c3eff2d2a2:/dev/nvme0n1p4 6d9df2ca-9cc3-498c-b144-04d67a5927c6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 10 02:46:41.834640 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.834563    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 02:46:41.837651 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.837632    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 02:46:41.839350 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.839234    2580 manager.go:216] Machine: {Timestamp:2026-09-10 02:46:41.838759739 +0000 UTC m=+0.370758284 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099256 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a1ebf0b4db17d560e2515b35c8a9c SystemUUID:ec2a1ebf-0b4d-b17d-560e-2515b35c8a9c BootID:902511a2-aa03-4ba9-a689-fa661e35f53e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d8:1d:9b:c8:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d8:1d:9b:c8:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:70:7d:4a:d5:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 02:46:41.839350 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.839338    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 02:46:41.839499 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.839433    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 02:46:41.843031 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.843010    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 02:46:41.843174 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.843034    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 02:46:41.843902 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.843892    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 02:46:41.843942 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.843906    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 02:46:41.843942 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.843913    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 02:46:41.843942 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.843926    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 02:46:41.845005 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.844995    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 02:46:41.848850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.848838    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 02:46:41.848966 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.848956    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 02:46:41.849006 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.848983    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 02:46:41.849006 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.848999    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 02:46:41.849108 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.849019    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 02:46:41.849108 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.849028    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 10 02:46:41.849196 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.849117    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 02:46:41.849661 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.849643    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 02:46:41.849734 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.849680    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 02:46:41.853246 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.853227    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 02:46:41.856263 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.856247    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 02:46:41.856350 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.856286    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 02:46:41.857091 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857077    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 02:46:41.857163 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857098    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 02:46:41.857163 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857107    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 02:46:41.857163 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857115    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 02:46:41.857163 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857126    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 02:46:41.857163 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857134    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 02:46:41.857344 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857181    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 02:46:41.857344 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857191    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 02:46:41.857344 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857201    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 02:46:41.857344 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857210    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 02:46:41.857344 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857222    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 02:46:41.857344 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.857234    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 02:46:41.857604 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.857585    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 02:46:41.857785 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.857767    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 02:46:41.858162 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.858151    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 02:46:41.858212 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.858165    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 02:46:41.861764 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.861744    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 02:46:41.862117 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.862104    2580 server.go:1268] "Started kubelet"
Sep 10 02:46:41.862161 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.862146    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 02:46:41.862249 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.862204    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 02:46:41.862303 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.862275    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 02:46:41.862977 ip-10-0-142-156 systemd[1]: Started Kubernetes Kubelet.
Sep 10 02:46:41.863765 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.863634    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 02:46:41.864263 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.864250    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 02:46:41.865021 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.864987    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7x65"
Sep 10 02:46:41.866786 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.866764    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 02:46:41.868338 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.867247    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-156.ec2.internal.18d3d5a9b52d698e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-156.ec2.internal,UID:ip-10-0-142-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-156.ec2.internal,},FirstTimestamp:2026-09-10 02:46:41.86206043 +0000 UTC m=+0.394058973,LastTimestamp:2026-09-10 02:46:41.86206043 +0000 UTC m=+0.394058973,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-156.ec2.internal,}"
Sep 10 02:46:41.870312 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.870297    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7x65"
Sep 10 02:46:41.870997 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.870981    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 02:46:41.871680 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.871641    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 02:46:41.871762 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.871746    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:41.872351 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872334    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 02:46:41.872351 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872336    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 02:46:41.872492 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872372    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 02:46:41.872492 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872430    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 02:46:41.872492 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872456    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 02:46:41.872622 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872512    2580 factory.go:55] Registering systemd factory
Sep 10 02:46:41.872622 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872545    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 10 02:46:41.872764 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872747    2580 factory.go:153] Registering CRI-O factory
Sep 10 02:46:41.872830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872768    2580 factory.go:223] Registration of the crio container factory successfully
Sep 10 02:46:41.872830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872820    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 02:46:41.872933 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872839    2580 factory.go:103] Registering Raw factory
Sep 10 02:46:41.872933 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.872854    2580 manager.go:1196] Started watching for new ooms in manager
Sep 10 02:46:41.873261 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.873243    2580 manager.go:349] Starting recovery of all containers
Sep 10 02:46:41.875796 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.875774    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 02:46:41.875905 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.875885    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 02:46:41.884915 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.883046    2580 manager.go:354] Recovery completed
Sep 10 02:46:41.888550 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.888537    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:41.890281 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.890267    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:41.890375 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.890296    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:41.890375 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.890311    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:41.890969 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.890863    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 02:46:41.890969 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.890875    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 02:46:41.890969 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.890863    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:41.890969 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.890894    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 02:46:41.891588 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.891571    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:41.891648 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.891598    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:41.891648 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.891611    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:41.891954 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.891942    2580 policy_none.go:49] "None policy: Start"
Sep 10 02:46:41.891984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.891957    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 02:46:41.891984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.891967    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 02:46:41.892913 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.892901    2580 policy_none.go:47] "Start"
Sep 10 02:46:41.919217 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.919200    2580 manager.go:337] "Starting Device Plugin manager"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.919255    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.919267    2580 server.go:87] "Starting device plugin registration server"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.919566    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.919579    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.919721    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.919791    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.919800    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.920378    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.920424    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.934953    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.936131    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.936173    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.936198    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:41.936235    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 02:46:41.938775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:41.937967    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:42.020023 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.019994    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:42.021000 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.020981    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:42.021114 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.021013    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:42.021114 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.021025    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:42.021114 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.021049    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.029932 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.029915    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.029982 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.029942    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-156.ec2.internal\": node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:42.036325 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.036305    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal"]
Sep 10 02:46:42.036410 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.036364    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:42.037191 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.037175    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:42.037266 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.037209    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:42.037266 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.037223    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:42.039431 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.039353    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:42.039498 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.039452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.039498 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.039489    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:42.040052 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.040034    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:42.040052 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.040044    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:42.040213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.040076    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:42.040213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.040096    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:42.040213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.040082    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:42.040213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.040164    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:42.041559 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.041538    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:42.042348 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.042330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.042442 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.042357    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:42.042978 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.042960    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:42.043058 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.042998    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:42.043058 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.043012    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:42.073597 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.073579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f22322537600d0788d094c2580f8d7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-156.ec2.internal\" (UID: \"32f22322537600d0788d094c2580f8d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.073698 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.073613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff8738e328686acd7be6bc98632632d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal\" (UID: \"ff8738e328686acd7be6bc98632632d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.073698 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.073631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff8738e328686acd7be6bc98632632d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal\" (UID: \"ff8738e328686acd7be6bc98632632d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.080738 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.080718    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-156.ec2.internal\" not found" node="ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.085001 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.084988    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-156.ec2.internal\" not found" node="ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.142269 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.142250    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:42.174395 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.174373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f22322537600d0788d094c2580f8d7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-156.ec2.internal\" (UID: \"32f22322537600d0788d094c2580f8d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.174523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.174400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff8738e328686acd7be6bc98632632d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal\" (UID: \"ff8738e328686acd7be6bc98632632d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.174523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.174415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff8738e328686acd7be6bc98632632d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal\" (UID: \"ff8738e328686acd7be6bc98632632d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.174523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.174436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32f22322537600d0788d094c2580f8d7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-156.ec2.internal\" (UID: \"32f22322537600d0788d094c2580f8d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.174523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.174484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff8738e328686acd7be6bc98632632d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal\" (UID: \"ff8738e328686acd7be6bc98632632d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.174523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.174487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff8738e328686acd7be6bc98632632d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal\" (UID: \"ff8738e328686acd7be6bc98632632d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.242533 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.242507    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:42.343147 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.343060    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:42.382663 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.382641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.388955 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:42.388934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:42.444214 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.444176    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:42.544697 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.544666    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:42.645177 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:42.645119    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:44.127824 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:44.127798    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:44.148215 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.148194    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:44.192280 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.192249    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 02:46:44.192692 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.192388    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 02:46:44.192692 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.192412    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 02:46:44.228575 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:44.228545    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:44.257791 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.257722    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 02:41:43 +0000 UTC" deadline="2028-03-29 14:31:57.111559098 +0000 UTC"
Sep 10 02:46:44.257791 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.257787    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13595h45m12.853775568s"
Sep 10 02:46:44.328827 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:44.328767    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:44.397752 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:44.397716    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f22322537600d0788d094c2580f8d7.slice/crio-52958f50c0b9835dea119d84f32eefea7183bdb3eee3e49a77dd0e1502fad057 WatchSource:0}: Error finding container 52958f50c0b9835dea119d84f32eefea7183bdb3eee3e49a77dd0e1502fad057: Status 404 returned error can't find the container with id 52958f50c0b9835dea119d84f32eefea7183bdb3eee3e49a77dd0e1502fad057
Sep 10 02:46:44.398083 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:44.398059    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8738e328686acd7be6bc98632632d0.slice/crio-2181a71b61429eac7144b73edc5fb1ef95f99a736bad045e9878cf995c37cbdd WatchSource:0}: Error finding container 2181a71b61429eac7144b73edc5fb1ef95f99a736bad045e9878cf995c37cbdd: Status 404 returned error can't find the container with id 2181a71b61429eac7144b73edc5fb1ef95f99a736bad045e9878cf995c37cbdd
Sep 10 02:46:44.405819 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.405804    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 02:46:44.429605 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:44.429583    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-156.ec2.internal\" not found"
Sep 10 02:46:44.440157 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.440142    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:44.454479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.454449    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:44.468028 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.468012    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 02:46:44.469188 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.469176    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal"
Sep 10 02:46:44.478349 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.478334    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 02:46:44.547376 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:44.547356    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:45.232789 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.232755    2580 apiserver.go:52] "Watching apiserver"
Sep 10 02:46:45.242329 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.242305    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 02:46:45.243685 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.243658    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-72mqv","kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal","openshift-cluster-node-tuning-operator/tuned-n2ps5","openshift-dns/node-resolver-6vbtf","openshift-image-registry/node-ca-mzhhr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal","openshift-multus/network-metrics-daemon-hs5b5","openshift-network-diagnostics/network-check-target-z7xf8","kube-system/global-pull-secret-syncer-rjz9f","kube-system/konnectivity-agent-s28lq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5","openshift-multus/multus-additional-cni-plugins-wg54z","openshift-multus/multus-wprt9","openshift-network-operator/iptables-alerter-h545f"]
Sep 10 02:46:45.246082 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.246056    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:45.246180 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.246144    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:46:45.248024 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.247996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.250092 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.250074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.250984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.250744    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 02:46:45.250984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.250745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 02:46:45.250984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.250839    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xpgld\""
Sep 10 02:46:45.250984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.250860    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 02:46:45.250984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.250906    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 02:46:45.252249 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.252232    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.252489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.252458    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 02:46:45.252586 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.252495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ltpr5\""
Sep 10 02:46:45.252751 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.252714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 02:46:45.253707 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.253693    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 02:46:45.254859 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.254416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.255029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.255014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 02:46:45.255070 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.255048    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 02:46:45.255274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.255260    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x4xgp\""
Sep 10 02:46:45.256779 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.256733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.256881 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.256859    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 02:46:45.257297 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.257105    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 02:46:45.257297 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.257147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 02:46:45.257297 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.257191    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t6556\""
Sep 10 02:46:45.258985 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.258969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.259058 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.259038    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:45.259373 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.259353    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 02:46:45.259469 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.259386    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 02:46:45.259526 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.259469    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 02:46:45.259526 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.259489    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c4x58\""
Sep 10 02:46:45.261641 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.261239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.263548 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.263520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:45.263642 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.263586    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:45.263762 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.263747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 02:46:45.265221 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.265130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rb9q9\""
Sep 10 02:46:45.265221 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.265129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 02:46:45.266566 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.266537    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 02:46:45.267233 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.266791    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 02:46:45.267349 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.265137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 02:46:45.267577 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.267550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-kubelet\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.267807 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.267757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-tuned\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.268228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.267662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.268228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvmc\" (UniqueName: \"kubernetes.io/projected/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-kube-api-access-7vvmc\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.268228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv94h\" (UniqueName: \"kubernetes.io/projected/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-kube-api-access-pv94h\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.268228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-cni-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-k8s-cni-cncf-io\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqqxx\" (UniqueName: \"kubernetes.io/projected/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-kube-api-access-kqqxx\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-var-lib-kubelet\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.268483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d9ef21-3103-4690-a438-6e727bd4b91e-hosts-file\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.268483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrnd4\" (UniqueName: \"kubernetes.io/projected/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-kube-api-access-rrnd4\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:45.268483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-socket-dir-parent\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9ef21-3103-4690-a438-6e727bd4b91e-tmp-dir\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc590b11-0eeb-40df-885c-5f5f5b0dd195-serviceca\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-system-cni-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-cni-binary-copy\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-cni-bin\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-cni-multus\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-run\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268649    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-lib-modules\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.268828 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.268813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 02:46:45.269279 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc590b11-0eeb-40df-885c-5f5f5b0dd195-host\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.269279 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvt6j\" (UniqueName: \"kubernetes.io/projected/fc590b11-0eeb-40df-885c-5f5f5b0dd195-kube-api-access-vvt6j\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.269381 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-hostroot\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-conf-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-multus-certs\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-etc-kubernetes\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysctl-conf\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnxb\" (UniqueName: \"kubernetes.io/projected/f2d9ef21-3103-4690-a438-6e727bd4b91e-kube-api-access-jwnxb\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-iptables-alerter-script\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.269886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-netns\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-daemon-config\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.269985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-systemd\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-sys\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-tmp\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-cnibin\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-kubernetes\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-host\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270304 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-host-slash\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.270705 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-modprobe-d\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270705 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysconfig\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270705 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysctl-d\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.270849 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.270730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.272985 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.272665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 02:46:45.272985 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.272803    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7px2q\""
Sep 10 02:46:45.272985 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.272856    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 02:46:45.273921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.273230    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 02:46:45.273921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.273394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.273921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.273480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 02:46:45.273921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.273757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4djz\""
Sep 10 02:46:45.274189 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.273931    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 02:46:45.276153 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.276137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 02:46:45.276257 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.276186    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26p67\""
Sep 10 02:46:45.276415 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.276397    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 02:46:45.283668 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.283645    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-czpn9"
Sep 10 02:46:45.290837 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.290805    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-czpn9"
Sep 10 02:46:45.322652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.322608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal" event={"ID":"32f22322537600d0788d094c2580f8d7","Type":"ContainerStarted","Data":"52958f50c0b9835dea119d84f32eefea7183bdb3eee3e49a77dd0e1502fad057"}
Sep 10 02:46:45.323701 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.323682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal" event={"ID":"ff8738e328686acd7be6bc98632632d0","Type":"ContainerStarted","Data":"2181a71b61429eac7144b73edc5fb1ef95f99a736bad045e9878cf995c37cbdd"}
Sep 10 02:46:45.355290 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.355271    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 02:46:45.370935 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.370907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.371033 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.370947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-run-netns\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371033 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.370971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371033 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d9ef21-3103-4690-a438-6e727bd4b91e-hosts-file\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.371033 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-socket-dir-parent\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-kubelet\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovn-node-metrics-cert\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9ef21-3103-4690-a438-6e727bd4b91e-tmp-dir\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d9ef21-3103-4690-a438-6e727bd4b91e-hosts-file\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-socket-dir-parent\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-cni-binary-copy\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-cni-bin\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-host\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovnkube-config\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvt6j\" (UniqueName: \"kubernetes.io/projected/fc590b11-0eeb-40df-885c-5f5f5b0dd195-kube-api-access-vvt6j\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-host\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9ef21-3103-4690-a438-6e727bd4b91e-tmp-dir\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69c7n\" (UniqueName: \"kubernetes.io/projected/e8ceac2a-9a49-4c9d-b922-d237e3940efa-kube-api-access-69c7n\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-cni-bin\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-ovn\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371538 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.371538    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b4ab09b-8102-437a-8a61-1333d7327dda-agent-certs\") pod \"konnectivity-agent-s28lq\" (UID: \"7b4ab09b-8102-437a-8a61-1333d7327dda\") " pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-netns\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.371604    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs podName:8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:45.871585783 +0000 UTC m=+3.021485473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs") pod "network-metrics-daemon-hs5b5" (UID: "8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-netns\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-daemon-config\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-cni-binary-copy\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-env-overrides\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-kubernetes\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-slash\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-kubernetes\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.371883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-cni-bin\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysctl-d\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42gw\" (UniqueName: \"kubernetes.io/projected/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-kube-api-access-l42gw\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.371974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-systemd-units\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-node-log\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysctl-d\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-kubelet\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-tuned\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-kubelet\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvmc\" (UniqueName: \"kubernetes.io/projected/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-kube-api-access-7vvmc\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-cni-netd\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-daemon-config\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovnkube-script-lib\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv94h\" (UniqueName: \"kubernetes.io/projected/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-kube-api-access-pv94h\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.372400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqqxx\" (UniqueName: \"kubernetes.io/projected/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-kube-api-access-kqqxx\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-var-lib-kubelet\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-device-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-var-lib-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpf2\" (UniqueName: \"kubernetes.io/projected/195235cd-1ec4-4e14-82e2-89d2fc8073b2-kube-api-access-prpf2\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-var-lib-kubelet\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrnd4\" (UniqueName: \"kubernetes.io/projected/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-kube-api-access-rrnd4\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-system-cni-dir\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc590b11-0eeb-40df-885c-5f5f5b0dd195-serviceca\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-system-cni-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-cni-multus\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-run\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372749    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-lib-modules\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-registration-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-system-cni-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/610b6131-5edf-4c48-a4d0-240d567c76c3-dbus\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-var-lib-cni-multus\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-run\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc590b11-0eeb-40df-885c-5f5f5b0dd195-host\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-hostroot\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc590b11-0eeb-40df-885c-5f5f5b0dd195-host\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.372999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-conf-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-lib-modules\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-multus-certs\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-hostroot\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-etc-kubernetes\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-multus-certs\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysctl-conf\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-etc-kubernetes\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-systemd\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc590b11-0eeb-40df-885c-5f5f5b0dd195-serviceca\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-conf-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-tmp\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.373824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-systemd\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnxb\" (UniqueName: \"kubernetes.io/projected/f2d9ef21-3103-4690-a438-6e727bd4b91e-kube-api-access-jwnxb\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-iptables-alerter-script\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-sys\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-sys-fs\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-systemd\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-log-socket\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b4ab09b-8102-437a-8a61-1333d7327dda-konnectivity-ca\") pod \"konnectivity-agent-s28lq\" (UID: \"7b4ab09b-8102-437a-8a61-1333d7327dda\") " pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-sys\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-cnibin\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-etc-selinux\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cnibin\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-cnibin\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.373782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-iptables-alerter-script\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-host-slash\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-host-slash\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.374507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-modprobe-d\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysconfig\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-socket-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysctl-conf\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-sysconfig\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-modprobe-d\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/610b6131-5edf-4c48-a4d0-240d567c76c3-kubelet-config\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-etc-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-cni-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-k8s-cni-cncf-io\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-host-run-k8s-cni-cncf-io\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.375205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.374774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-multus-cni-dir\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.377428 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.377400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-tmp\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.377428 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.377417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-etc-tuned\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.383439 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.383404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvt6j\" (UniqueName: \"kubernetes.io/projected/fc590b11-0eeb-40df-885c-5f5f5b0dd195-kube-api-access-vvt6j\") pod \"node-ca-mzhhr\" (UID: \"fc590b11-0eeb-40df-885c-5f5f5b0dd195\") " pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.384381 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.384356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv94h\" (UniqueName: \"kubernetes.io/projected/c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127-kube-api-access-pv94h\") pod \"iptables-alerter-h545f\" (UID: \"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127\") " pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.384858 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.384836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvmc\" (UniqueName: \"kubernetes.io/projected/36e04e65-7922-4e07-8d23-cac3b1bbcf4e-kube-api-access-7vvmc\") pod \"tuned-n2ps5\" (UID: \"36e04e65-7922-4e07-8d23-cac3b1bbcf4e\") " pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.385099 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.385077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqqxx\" (UniqueName: \"kubernetes.io/projected/e2d84b3d-2fbe-47f2-9552-05c33d77fb67-kube-api-access-kqqxx\") pod \"multus-wprt9\" (UID: \"e2d84b3d-2fbe-47f2-9552-05c33d77fb67\") " pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.385552 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.385533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnxb\" (UniqueName: \"kubernetes.io/projected/f2d9ef21-3103-4690-a438-6e727bd4b91e-kube-api-access-jwnxb\") pod \"node-resolver-6vbtf\" (UID: \"f2d9ef21-3103-4690-a438-6e727bd4b91e\") " pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.386331 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.386311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrnd4\" (UniqueName: \"kubernetes.io/projected/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-kube-api-access-rrnd4\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:45.475746 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovn-node-metrics-cert\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.475922 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovnkube-config\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.475922 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69c7n\" (UniqueName: \"kubernetes.io/projected/e8ceac2a-9a49-4c9d-b922-d237e3940efa-kube-api-access-69c7n\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.475922 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-ovn\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.475922 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b4ab09b-8102-437a-8a61-1333d7327dda-agent-certs\") pod \"konnectivity-agent-s28lq\" (UID: \"7b4ab09b-8102-437a-8a61-1333d7327dda\") " pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.475922 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.475922 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-ovn\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-env-overrides\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-slash\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.475992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-cni-bin\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42gw\" (UniqueName: \"kubernetes.io/projected/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-kube-api-access-l42gw\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-systemd-units\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-node-log\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-cni-netd\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovnkube-script-lib\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-slash\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-systemd-units\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-node-log\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-cni-netd\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-device-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovnkube-config\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-var-lib-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-env-overrides\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-var-lib-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpf2\" (UniqueName: \"kubernetes.io/projected/195235cd-1ec4-4e14-82e2-89d2fc8073b2-kube-api-access-prpf2\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-device-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-cni-bin\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-system-cni-dir\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-system-cni-dir\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-registration-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/610b6131-5edf-4c48-a4d0-240d567c76c3-dbus\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.476876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-sys-fs\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-systemd\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-registration-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-log-socket\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/610b6131-5edf-4c48-a4d0-240d567c76c3-dbus\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-log-socket\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b4ab09b-8102-437a-8a61-1333d7327dda-konnectivity-ca\") pod \"konnectivity-agent-s28lq\" (UID: \"7b4ab09b-8102-437a-8a61-1333d7327dda\") " pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-systemd\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-etc-selinux\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovnkube-script-lib\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cnibin\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.476959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-sys-fs\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-cnibin\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-socket-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.477672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/610b6131-5edf-4c48-a4d0-240d567c76c3-kubelet-config\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-etc-selinux\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-socket-dir\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/610b6131-5edf-4c48-a4d0-240d567c76c3-kubelet-config\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-etc-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.477247    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-etc-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8ceac2a-9a49-4c9d-b922-d237e3940efa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-run-netns\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.477302    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret podName:610b6131-5edf-4c48-a4d0-240d567c76c3 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:45.977283013 +0000 UTC m=+3.127182690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret") pod "global-pull-secret-syncer-rjz9f" (UID: "610b6131-5edf-4c48-a4d0-240d567c76c3") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-run-netns\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-run-openvswitch\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.478501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.479274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-kubelet\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.479274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b4ab09b-8102-437a-8a61-1333d7327dda-konnectivity-ca\") pod \"konnectivity-agent-s28lq\" (UID: \"7b4ab09b-8102-437a-8a61-1333d7327dda\") " pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.479274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.479274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195235cd-1ec4-4e14-82e2-89d2fc8073b2-host-kubelet\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.479274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.477667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8ceac2a-9a49-4c9d-b922-d237e3940efa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.479274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.478556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195235cd-1ec4-4e14-82e2-89d2fc8073b2-ovn-node-metrics-cert\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.479274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.478580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b4ab09b-8102-437a-8a61-1333d7327dda-agent-certs\") pod \"konnectivity-agent-s28lq\" (UID: \"7b4ab09b-8102-437a-8a61-1333d7327dda\") " pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.485424 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.485358    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:45.485424 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.485382    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:45.485424 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.485394    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hdnnx for pod openshift-network-diagnostics/network-check-target-z7xf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:45.485624 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.485456    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx podName:1a798373-1bde-4671-83d5-0c1e6f678c33 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:45.985440895 +0000 UTC m=+3.135340578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hdnnx" (UniqueName: "kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx") pod "network-check-target-z7xf8" (UID: "1a798373-1bde-4671-83d5-0c1e6f678c33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:45.488034 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.488008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpf2\" (UniqueName: \"kubernetes.io/projected/195235cd-1ec4-4e14-82e2-89d2fc8073b2-kube-api-access-prpf2\") pod \"ovnkube-node-72mqv\" (UID: \"195235cd-1ec4-4e14-82e2-89d2fc8073b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.488588 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.488566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42gw\" (UniqueName: \"kubernetes.io/projected/6b6b4338-eab1-460d-ae5a-0bc14fbb0864-kube-api-access-l42gw\") pod \"aws-ebs-csi-driver-node-fdbv5\" (UID: \"6b6b4338-eab1-460d-ae5a-0bc14fbb0864\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.489371 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.489349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69c7n\" (UniqueName: \"kubernetes.io/projected/e8ceac2a-9a49-4c9d-b922-d237e3940efa-kube-api-access-69c7n\") pod \"multus-additional-cni-plugins-wg54z\" (UID: \"e8ceac2a-9a49-4c9d-b922-d237e3940efa\") " pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.557239 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.557210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wprt9"
Sep 10 02:46:45.564475 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.564455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n2ps5"
Sep 10 02:46:45.576983 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.576964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6vbtf"
Sep 10 02:46:45.583284 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.583268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzhhr"
Sep 10 02:46:45.591821 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.591802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h545f"
Sep 10 02:46:45.599530 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.599516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:46:45.606943 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.606925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:46:45.614430 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.614410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5"
Sep 10 02:46:45.621067 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.621049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wg54z"
Sep 10 02:46:45.793396 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.793313    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:45.880072 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.880031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:45.880273 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.880182    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:45.880273 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.880256    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs podName:8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:46.880233816 +0000 UTC m=+4.030133509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs") pod "network-metrics-daemon-hs5b5" (UID: "8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:45.981191 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:45.981146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:45.981363 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.981335    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:45.981429 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:45.981416    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret podName:610b6131-5edf-4c48-a4d0-240d567c76c3 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:46.981393374 +0000 UTC m=+4.131293072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret") pod "global-pull-secret-syncer-rjz9f" (UID: "610b6131-5edf-4c48-a4d0-240d567c76c3") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:46.082472 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.082402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:46.082609 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.082541    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:46.082609 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.082557    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:46.082609 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.082567    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hdnnx for pod openshift-network-diagnostics/network-check-target-z7xf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:46.082736 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.082616    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx podName:1a798373-1bde-4671-83d5-0c1e6f678c33 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:47.082600194 +0000 UTC m=+4.232499890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdnnx" (UniqueName: "kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx") pod "network-check-target-z7xf8" (UID: "1a798373-1bde-4671-83d5-0c1e6f678c33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:46.282715 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.282692    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d9ef21_3103_4690_a438_6e727bd4b91e.slice/crio-a42bba57983e745df9f27e6f9f94ad0ac2a26916cca39b65f45979459a5f4f87 WatchSource:0}: Error finding container a42bba57983e745df9f27e6f9f94ad0ac2a26916cca39b65f45979459a5f4f87: Status 404 returned error can't find the container with id a42bba57983e745df9f27e6f9f94ad0ac2a26916cca39b65f45979459a5f4f87
Sep 10 02:46:46.284217 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.283779    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e04e65_7922_4e07_8d23_cac3b1bbcf4e.slice/crio-b5f21ae2cb24ff83ed290087012bdd087717488e214d4c931176c0a75ac6b48e WatchSource:0}: Error finding container b5f21ae2cb24ff83ed290087012bdd087717488e214d4c931176c0a75ac6b48e: Status 404 returned error can't find the container with id b5f21ae2cb24ff83ed290087012bdd087717488e214d4c931176c0a75ac6b48e
Sep 10 02:46:46.288530 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.288507    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d84b3d_2fbe_47f2_9552_05c33d77fb67.slice/crio-7fcfc6ab176405dd992009869c002d879fc8b11a1e32b0bb4e7c42fb1ad67127 WatchSource:0}: Error finding container 7fcfc6ab176405dd992009869c002d879fc8b11a1e32b0bb4e7c42fb1ad67127: Status 404 returned error can't find the container with id 7fcfc6ab176405dd992009869c002d879fc8b11a1e32b0bb4e7c42fb1ad67127
Sep 10 02:46:46.289216 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.289191    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc951b2bb_5b7c_4cc3_94f4_0c88dbfaa127.slice/crio-d64fcaf84bf0031456fd93edd74f920ccc3869895ff96bf5e2402d730aaa2249 WatchSource:0}: Error finding container d64fcaf84bf0031456fd93edd74f920ccc3869895ff96bf5e2402d730aaa2249: Status 404 returned error can't find the container with id d64fcaf84bf0031456fd93edd74f920ccc3869895ff96bf5e2402d730aaa2249
Sep 10 02:46:46.291437 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.291415    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc590b11_0eeb_40df_885c_5f5f5b0dd195.slice/crio-112d9c41d3d7651a48b3a18a24f89132e25d28d6b163dcbae8b057fefd8ddacc WatchSource:0}: Error finding container 112d9c41d3d7651a48b3a18a24f89132e25d28d6b163dcbae8b057fefd8ddacc: Status 404 returned error can't find the container with id 112d9c41d3d7651a48b3a18a24f89132e25d28d6b163dcbae8b057fefd8ddacc
Sep 10 02:46:46.292346 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.292266    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195235cd_1ec4_4e14_82e2_89d2fc8073b2.slice/crio-6f654d7afb5bfe95d55e517a4d21756eb617c81869ead9977b45bd16605183d0 WatchSource:0}: Error finding container 6f654d7afb5bfe95d55e517a4d21756eb617c81869ead9977b45bd16605183d0: Status 404 returned error can't find the container with id 6f654d7afb5bfe95d55e517a4d21756eb617c81869ead9977b45bd16605183d0
Sep 10 02:46:46.293091 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.292980    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6b4338_eab1_460d_ae5a_0bc14fbb0864.slice/crio-9d78a60c1c16280d364713cc67ca567d71828d59450ca6c9d3e2c9ed098b849d WatchSource:0}: Error finding container 9d78a60c1c16280d364713cc67ca567d71828d59450ca6c9d3e2c9ed098b849d: Status 404 returned error can't find the container with id 9d78a60c1c16280d364713cc67ca567d71828d59450ca6c9d3e2c9ed098b849d
Sep 10 02:46:46.293914 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.293633    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:41:45 +0000 UTC" deadline="2028-05-14 07:46:15.88016456 +0000 UTC"
Sep 10 02:46:46.293914 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.293662    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14692h59m29.586507254s"
Sep 10 02:46:46.295739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.294505    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ceac2a_9a49_4c9d_b922_d237e3940efa.slice/crio-4e331e59505c54725fd0e504134a35acae877b0e67c78e6ea1e3afe080ca5643 WatchSource:0}: Error finding container 4e331e59505c54725fd0e504134a35acae877b0e67c78e6ea1e3afe080ca5643: Status 404 returned error can't find the container with id 4e331e59505c54725fd0e504134a35acae877b0e67c78e6ea1e3afe080ca5643
Sep 10 02:46:46.295739 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:46:46.294739    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4ab09b_8102_437a_8a61_1333d7327dda.slice/crio-8b9810b75446c2fd77aa151e7008e4f4dc1896fc49dab6c71c0e890d9d099791 WatchSource:0}: Error finding container 8b9810b75446c2fd77aa151e7008e4f4dc1896fc49dab6c71c0e890d9d099791: Status 404 returned error can't find the container with id 8b9810b75446c2fd77aa151e7008e4f4dc1896fc49dab6c71c0e890d9d099791
Sep 10 02:46:46.318952 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.318929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:46.319057 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.319039    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:46.328597 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.328572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h545f" event={"ID":"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127","Type":"ContainerStarted","Data":"d64fcaf84bf0031456fd93edd74f920ccc3869895ff96bf5e2402d730aaa2249"}
Sep 10 02:46:46.331496 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.331476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s28lq" event={"ID":"7b4ab09b-8102-437a-8a61-1333d7327dda","Type":"ContainerStarted","Data":"8b9810b75446c2fd77aa151e7008e4f4dc1896fc49dab6c71c0e890d9d099791"}
Sep 10 02:46:46.335952 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.335892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerStarted","Data":"4e331e59505c54725fd0e504134a35acae877b0e67c78e6ea1e3afe080ca5643"}
Sep 10 02:46:46.337071 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.337051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5" event={"ID":"6b6b4338-eab1-460d-ae5a-0bc14fbb0864","Type":"ContainerStarted","Data":"9d78a60c1c16280d364713cc67ca567d71828d59450ca6c9d3e2c9ed098b849d"}
Sep 10 02:46:46.338205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.338182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"6f654d7afb5bfe95d55e517a4d21756eb617c81869ead9977b45bd16605183d0"}
Sep 10 02:46:46.339297 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.339273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wprt9" event={"ID":"e2d84b3d-2fbe-47f2-9552-05c33d77fb67","Type":"ContainerStarted","Data":"7fcfc6ab176405dd992009869c002d879fc8b11a1e32b0bb4e7c42fb1ad67127"}
Sep 10 02:46:46.340516 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.340494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n2ps5" event={"ID":"36e04e65-7922-4e07-8d23-cac3b1bbcf4e","Type":"ContainerStarted","Data":"b5f21ae2cb24ff83ed290087012bdd087717488e214d4c931176c0a75ac6b48e"}
Sep 10 02:46:46.341542 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.341521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6vbtf" event={"ID":"f2d9ef21-3103-4690-a438-6e727bd4b91e","Type":"ContainerStarted","Data":"a42bba57983e745df9f27e6f9f94ad0ac2a26916cca39b65f45979459a5f4f87"}
Sep 10 02:46:46.342623 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.342604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzhhr" event={"ID":"fc590b11-0eeb-40df-885c-5f5f5b0dd195","Type":"ContainerStarted","Data":"112d9c41d3d7651a48b3a18a24f89132e25d28d6b163dcbae8b057fefd8ddacc"}
Sep 10 02:46:46.887546 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.887506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:46.887701 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.887648    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:46.887760 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.887705    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs podName:8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:48.887686784 +0000 UTC m=+6.037586465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs") pod "network-metrics-daemon-hs5b5" (UID: "8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:46.990101 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:46.988669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:46.990101 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.988826    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:46.990101 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:46.988881    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret podName:610b6131-5edf-4c48-a4d0-240d567c76c3 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:48.988864234 +0000 UTC m=+6.138763916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret") pod "global-pull-secret-syncer-rjz9f" (UID: "610b6131-5edf-4c48-a4d0-240d567c76c3") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:47.089019 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.088984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:47.089207 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:47.089191    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:47.089267 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:47.089219    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:47.089267 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:47.089232    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hdnnx for pod openshift-network-diagnostics/network-check-target-z7xf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:47.089383 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:47.089285    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx podName:1a798373-1bde-4671-83d5-0c1e6f678c33 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:49.089267291 +0000 UTC m=+6.239166984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdnnx" (UniqueName: "kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx") pod "network-check-target-z7xf8" (UID: "1a798373-1bde-4671-83d5-0c1e6f678c33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:47.294777 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.294628    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:41:45 +0000 UTC" deadline="2028-06-23 07:24:42.051306376 +0000 UTC"
Sep 10 02:46:47.294777 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.294694    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15652h37m54.75661657s"
Sep 10 02:46:47.321582 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.321529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:47.321713 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:47.321643    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:47.322082 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.322034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:47.322176 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:47.322144    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:46:47.331003 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.330978    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:47.363120 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.363067    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff8738e328686acd7be6bc98632632d0" containerID="d6027cf1aa30815f8d9f9ebf92da62a2b73e5ce9e6cc0a31b1918e62ded9c5e0" exitCode=0
Sep 10 02:46:47.364027 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.363155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal" event={"ID":"ff8738e328686acd7be6bc98632632d0","Type":"ContainerDied","Data":"d6027cf1aa30815f8d9f9ebf92da62a2b73e5ce9e6cc0a31b1918e62ded9c5e0"}
Sep 10 02:46:47.389903 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:47.389871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal" event={"ID":"32f22322537600d0788d094c2580f8d7","Type":"ContainerStarted","Data":"fe763ada3637c7e86ea2982ab223984f0338b985bc28fee5a23be78879a2407b"}
Sep 10 02:46:48.319223 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:48.318966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:48.319637 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:48.319319    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:48.414066 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:48.414034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal" event={"ID":"ff8738e328686acd7be6bc98632632d0","Type":"ContainerStarted","Data":"e4436ffb192c62187aa47f00ae2a512baf026030e00d6074cf9d5c4b5df31a19"}
Sep 10 02:46:48.428334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:48.428280    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-156.ec2.internal" podStartSLOduration=4.428257995 podStartE2EDuration="4.428257995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:47.405159625 +0000 UTC m=+4.555059324" watchObservedRunningTime="2026-09-10 02:46:48.428257995 +0000 UTC m=+5.578157695"
Sep 10 02:46:48.904678 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:48.904641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:48.904867 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:48.904799    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:48.904867 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:48.904865    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs podName:8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:52.904845534 +0000 UTC m=+10.054745225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs") pod "network-metrics-daemon-hs5b5" (UID: "8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:49.005985 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:49.005942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:49.006210 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.006190    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:49.006289 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.006261    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret podName:610b6131-5edf-4c48-a4d0-240d567c76c3 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:53.006242872 +0000 UTC m=+10.156142562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret") pod "global-pull-secret-syncer-rjz9f" (UID: "610b6131-5edf-4c48-a4d0-240d567c76c3") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:49.108080 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:49.107383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:49.108080 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.107580    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:49.108080 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.107599    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:49.108080 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.107612    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hdnnx for pod openshift-network-diagnostics/network-check-target-z7xf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:49.108080 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.107666    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx podName:1a798373-1bde-4671-83d5-0c1e6f678c33 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:53.107648308 +0000 UTC m=+10.257547988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdnnx" (UniqueName: "kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx") pod "network-check-target-z7xf8" (UID: "1a798373-1bde-4671-83d5-0c1e6f678c33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:49.318854 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:49.318779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:49.319009 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.318946    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:46:49.319369 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:49.319339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:49.319725 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:49.319440    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:50.318536 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:50.318500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:50.318705 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:50.318636    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:51.319984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:51.319950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:51.320534 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:51.320101    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:46:51.320534 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:51.320179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:51.320534 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:51.320298    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:52.319349 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:52.319240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:52.319503 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:52.319371    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:52.938440 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:52.938403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:52.938907 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:52.938598    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:52.938907 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:52.938657    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs podName:8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:00.938638143 +0000 UTC m=+18.088537823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs") pod "network-metrics-daemon-hs5b5" (UID: "8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:53.039523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:53.039483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:53.039679 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.039644    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:53.039735 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.039709    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret podName:610b6131-5edf-4c48-a4d0-240d567c76c3 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:01.039690527 +0000 UTC m=+18.189590207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret") pod "global-pull-secret-syncer-rjz9f" (UID: "610b6131-5edf-4c48-a4d0-240d567c76c3") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:53.140295 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:53.140188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:53.140454 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.140319    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:53.140454 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.140346    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:53.140454 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.140360    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hdnnx for pod openshift-network-diagnostics/network-check-target-z7xf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:53.140454 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.140423    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx podName:1a798373-1bde-4671-83d5-0c1e6f678c33 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:01.140402917 +0000 UTC m=+18.290302600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdnnx" (UniqueName: "kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx") pod "network-check-target-z7xf8" (UID: "1a798373-1bde-4671-83d5-0c1e6f678c33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:53.320929 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:53.320853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:53.321071 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.320960    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:53.321363 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:53.321344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:53.321473 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:53.321450    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:46:54.318902 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:54.318813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:54.319340 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:54.318950    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:55.319059 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:55.318756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:55.319059 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:55.318905    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:46:55.319059 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:55.318946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:55.319059 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:55.319023    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:56.319407 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:56.319387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:56.319656 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:56.319484    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:57.318747 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:57.318710    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:57.318910 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:57.318717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:57.318910 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:57.318830    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:46:57.318910 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:57.318899    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:58.319123 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:58.319093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:46:58.319563 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:58.319501    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:46:59.318846 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:59.318675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:46:59.318846 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:59.318795    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:46:59.318846 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:46:59.318846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:46:59.319127 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:46:59.318952    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:00.319505 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:00.319476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:00.319983 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:00.319588    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:00.993539 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:00.993502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:00.993704 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:00.993640    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:00.993704 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:00.993696    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs podName:8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:16.993680155 +0000 UTC m=+34.143579831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs") pod "network-metrics-daemon-hs5b5" (UID: "8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:01.094406 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:01.094374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:01.094569 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.094502    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:01.094569 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.094556    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret podName:610b6131-5edf-4c48-a4d0-240d567c76c3 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.094542547 +0000 UTC m=+34.244442223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret") pod "global-pull-secret-syncer-rjz9f" (UID: "610b6131-5edf-4c48-a4d0-240d567c76c3") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:01.195599 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:01.195562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:01.195771 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.195718    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:47:01.195771 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.195736    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:47:01.195771 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.195747    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hdnnx for pod openshift-network-diagnostics/network-check-target-z7xf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:01.195901 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.195801    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx podName:1a798373-1bde-4671-83d5-0c1e6f678c33 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.195783904 +0000 UTC m=+34.345683582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdnnx" (UniqueName: "kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx") pod "network-check-target-z7xf8" (UID: "1a798373-1bde-4671-83d5-0c1e6f678c33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:01.318750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:01.318671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:01.318750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:01.318713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:01.318960 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.318809    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:01.318960 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:01.318942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:02.319281 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:02.319249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:02.319729 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:02.319369    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:03.319408 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:03.319381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:03.319839 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:03.319493    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:03.319839 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:03.319588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:03.319839 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:03.319707    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:04.319711 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.319486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:04.320336 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:04.319807    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:04.445134 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.445110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"83c4ad3428e1303d33746d46ea3248d84f363a65ba1686531f859a02f079268d"}
Sep 10 02:47:04.445247 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.445194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"216d41b6c23cfca089b060a321a79730c43e559590cb34155bdf2ffb712c968a"}
Sep 10 02:47:04.445247 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.445225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"cad149fa7cb00bb286cb521051b8aa94b2306cebdf19e7e868283d566f13ca1c"}
Sep 10 02:47:04.445324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.445254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"d9748e2a5a9401c547034d836dcbdc680e5d0dff23339d6784b478efaaf9710f"}
Sep 10 02:47:04.445324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.445282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"0e9ff8ab8a302432fcf16360cab4d7ed107e9f6b2a4ee0f9df1921c02783d810"}
Sep 10 02:47:04.446483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.446457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wprt9" event={"ID":"e2d84b3d-2fbe-47f2-9552-05c33d77fb67","Type":"ContainerStarted","Data":"29e5835e05bdfd5447f5bdd5a26dd7fdb53a729d61d10b3f198c34a5fd40f2ff"}
Sep 10 02:47:04.447886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.447864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n2ps5" event={"ID":"36e04e65-7922-4e07-8d23-cac3b1bbcf4e","Type":"ContainerStarted","Data":"6dd5d3d8fc2bb12db9ba9866f775dd895f74b48d5709399f7928e75953613783"}
Sep 10 02:47:04.449322 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.449298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6vbtf" event={"ID":"f2d9ef21-3103-4690-a438-6e727bd4b91e","Type":"ContainerStarted","Data":"3328a66575d760635304ece8762dc01b7f4a755a101eff4308866c6597f4b7af"}
Sep 10 02:47:04.450709 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.450685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzhhr" event={"ID":"fc590b11-0eeb-40df-885c-5f5f5b0dd195","Type":"ContainerStarted","Data":"25053d30b94881f15f05bcaded6f9f2e67028b17234ca1ffc867a524d56d86b8"}
Sep 10 02:47:04.452388 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.452364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s28lq" event={"ID":"7b4ab09b-8102-437a-8a61-1333d7327dda","Type":"ContainerStarted","Data":"2b27a6dae66ec7bac05b4f993c864af93f00fbe489139b3b15132ed227ecdc21"}
Sep 10 02:47:04.453923 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.453901    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8ceac2a-9a49-4c9d-b922-d237e3940efa" containerID="dfd27663691e8f30c522bac2dee9d35e31ffad6d3a0f378d9ea77697c9dc44fd" exitCode=0
Sep 10 02:47:04.454023 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.454003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerDied","Data":"dfd27663691e8f30c522bac2dee9d35e31ffad6d3a0f378d9ea77697c9dc44fd"}
Sep 10 02:47:04.455611 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.455588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5" event={"ID":"6b6b4338-eab1-460d-ae5a-0bc14fbb0864","Type":"ContainerStarted","Data":"63cd7d3d132b36bf7b47cd15f32d0eaaeb46eb1f86b522265040411e1d4cfafc"}
Sep 10 02:47:04.471691 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.471642    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wprt9" podStartSLOduration=3.837783831 podStartE2EDuration="21.471624108s" totalImagesPullingTime="17.633840277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:04.471265252 +0000 UTC m=+21.621164951" watchObservedRunningTime="2026-09-10 02:47:04.471624108 +0000 UTC m=+21.621523807"
Sep 10 02:47:04.472114 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.472087    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-156.ec2.internal" podStartSLOduration=20.472078184 podStartE2EDuration="20.472078184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:46:48.429090254 +0000 UTC m=+5.578989953" watchObservedRunningTime="2026-09-10 02:47:04.472078184 +0000 UTC m=+21.621977883"
Sep 10 02:47:04.503004 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.502963    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mzhhr" podStartSLOduration=3.929281977 podStartE2EDuration="21.502948989s" totalImagesPullingTime="17.573667012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:04.487090466 +0000 UTC m=+21.636990165" watchObservedRunningTime="2026-09-10 02:47:04.502948989 +0000 UTC m=+21.652848686"
Sep 10 02:47:04.503146 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.503125    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s28lq" podStartSLOduration=8.084695098 podStartE2EDuration="21.503121016s" totalImagesPullingTime="13.418425918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:04.502738048 +0000 UTC m=+21.652637746" watchObservedRunningTime="2026-09-10 02:47:04.503121016 +0000 UTC m=+21.653020714"
Sep 10 02:47:04.519614 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.519543    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n2ps5" podStartSLOduration=3.939081141 podStartE2EDuration="21.519532034s" totalImagesPullingTime="17.580450893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:04.519513369 +0000 UTC m=+21.669413067" watchObservedRunningTime="2026-09-10 02:47:04.519532034 +0000 UTC m=+21.669431732"
Sep 10 02:47:04.537589 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:04.537554    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6vbtf" podStartSLOduration=3.955136459 podStartE2EDuration="21.537541627s" totalImagesPullingTime="17.582405168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:04.536899492 +0000 UTC m=+21.686799191" watchObservedRunningTime="2026-09-10 02:47:04.537541627 +0000 UTC m=+21.687441324"
Sep 10 02:47:05.318883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:05.318675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:05.319002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:05.318763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:05.319067 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:05.318997    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:05.319124 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:05.319085    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:05.328861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:05.328837    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 02:47:05.460058 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:05.459980    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h545f" event={"ID":"c951b2bb-5b7c-4cc3-94f4-0c88dbfaa127","Type":"ContainerStarted","Data":"9d564a7fec8b5cff6662d940a992c4759c6e4153db51c8876e53d6ef161e51d9"}
Sep 10 02:47:05.462193 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:05.462096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5" event={"ID":"6b6b4338-eab1-460d-ae5a-0bc14fbb0864","Type":"ContainerStarted","Data":"d88310aedc56c3c42aa2bc72fd14a2fe499a01c92c3d4ba89d295b1823df0eba"}
Sep 10 02:47:05.466034 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:05.465918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"30e432f84eb40cb2b9481e502795c5553e35f523518d3b1dc5727403217c29e7"}
Sep 10 02:47:05.475410 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:05.475362    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h545f" podStartSLOduration=4.899017742 podStartE2EDuration="22.475345965s" totalImagesPullingTime="17.576328223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:05.47533596 +0000 UTC m=+22.625235658" watchObservedRunningTime="2026-09-10 02:47:05.475345965 +0000 UTC m=+22.625245663"
Sep 10 02:47:06.312323 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:06.312227    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T02:47:05.328855221Z","UUID":"68499dcb-9510-4d72-9f01-594a5e1bba13","Handler":null,"Name":"","Endpoint":""}
Sep 10 02:47:06.313884 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:06.313862    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 02:47:06.313996 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:06.313897    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 02:47:06.318523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:06.318499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:06.318662 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:06.318639    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:07.319397 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:07.319369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:07.319862 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:07.319476    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:07.319862 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:07.319525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:07.319862 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:07.319608    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:07.473373 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:07.473334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5" event={"ID":"6b6b4338-eab1-460d-ae5a-0bc14fbb0864","Type":"ContainerStarted","Data":"543ee346dae037bf60f1c84fb687d5918b0541fa36c28ce4bd21da5e562468bd"}
Sep 10 02:47:07.477023 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:07.476994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"1d6126133b6a7d6287d1db6f905e3350846a8188844976577457e6f0decf1685"}
Sep 10 02:47:07.493266 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:07.493221    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fdbv5" podStartSLOduration=4.709204724 podStartE2EDuration="24.493204218s" totalImagesPullingTime="19.783999494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:07.492681347 +0000 UTC m=+24.642581047" watchObservedRunningTime="2026-09-10 02:47:07.493204218 +0000 UTC m=+24.643103912"
Sep 10 02:47:07.652325 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:07.652294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:47:07.653367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:07.653342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:47:08.319268 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:08.319237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:08.319432 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:08.319338    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:08.483560 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:08.483455    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:47:08.483817 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:08.483798    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s28lq"
Sep 10 02:47:09.319567 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.319336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:09.320129 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.319381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:09.320129 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:09.319692    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:09.320129 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:09.319791    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:09.484282 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.484251    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8ceac2a-9a49-4c9d-b922-d237e3940efa" containerID="0e0a0da18d9954e8245711ff0925e163f0420d3e56821b3ad8741a3a798e1930" exitCode=0
Sep 10 02:47:09.484442 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.484303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerDied","Data":"0e0a0da18d9954e8245711ff0925e163f0420d3e56821b3ad8741a3a798e1930"}
Sep 10 02:47:09.488414 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.488391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" event={"ID":"195235cd-1ec4-4e14-82e2-89d2fc8073b2","Type":"ContainerStarted","Data":"475ac537693b56a4cfe2fdc3aca127fead841e5b1e5b0528696a61f6f193d637"}
Sep 10 02:47:09.488648 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.488621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:47:09.488736 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.488679    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:47:09.503138 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.503084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:47:09.531925 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:09.531890    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv" podStartSLOduration=8.896485905 podStartE2EDuration="26.531877418s" totalImagesPullingTime="17.635391513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:09.531419587 +0000 UTC m=+26.681319286" watchObservedRunningTime="2026-09-10 02:47:09.531877418 +0000 UTC m=+26.681777144"
Sep 10 02:47:10.318875 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.318848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:10.319112 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:10.318963    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:10.491580 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.491428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:47:10.507639 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.507608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:47:10.939829 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.939803    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rjz9f"]
Sep 10 02:47:10.940005 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.939905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:10.940005 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:10.939983    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:10.942691 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.942664    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7xf8"]
Sep 10 02:47:10.942823 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.942762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:10.942872 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:10.942839    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:10.943456 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.943437    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hs5b5"]
Sep 10 02:47:10.943568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:10.943525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:10.943612 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:10.943600    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:11.494915 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:11.494886    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8ceac2a-9a49-4c9d-b922-d237e3940efa" containerID="8fa5cc2949565b9cbf19f5a00b35e4584b884219f4e38aa1e593b38e6ae32c31" exitCode=0
Sep 10 02:47:11.495366 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:11.494977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerDied","Data":"8fa5cc2949565b9cbf19f5a00b35e4584b884219f4e38aa1e593b38e6ae32c31"}
Sep 10 02:47:12.318889 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:12.318859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:12.319041 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:12.318952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:12.319041 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:12.318956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:12.319041 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:12.319032    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:12.319209 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:12.318954    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:12.319209 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:12.319116    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:12.499408 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:12.499333    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8ceac2a-9a49-4c9d-b922-d237e3940efa" containerID="c70a645d3762045ed52c39ab9284ff8300e7e47fdf72c49308adc682f7f315b7" exitCode=0
Sep 10 02:47:12.499752 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:12.499425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerDied","Data":"c70a645d3762045ed52c39ab9284ff8300e7e47fdf72c49308adc682f7f315b7"}
Sep 10 02:47:14.318521 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:14.318485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:14.318904 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:14.318534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:14.318904 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:14.318534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:14.318904 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:14.318608    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:14.318904 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:14.318704    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:14.318904 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:14.318796    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:16.319251 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.319212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:16.319773 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.319212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:16.319773 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:16.319329    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjz9f" podUID="610b6131-5edf-4c48-a4d0-240d567c76c3"
Sep 10 02:47:16.319773 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.319344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:16.319773 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:16.319438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hs5b5" podUID="8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5"
Sep 10 02:47:16.319773 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:16.319518    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7xf8" podUID="1a798373-1bde-4671-83d5-0c1e6f678c33"
Sep 10 02:47:16.680705 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.680541    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-156.ec2.internal" event="NodeReady"
Sep 10 02:47:16.680838 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.680794    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 02:47:16.728098 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.728065    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lrh84"]
Sep 10 02:47:16.738875 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.738849    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v7vms"]
Sep 10 02:47:16.738993 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.738968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.741837 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.741813    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 02:47:16.741969 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.741937    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 02:47:16.742141 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.742127    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 02:47:16.742246 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.742197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fq77l\""
Sep 10 02:47:16.742246 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.742209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 02:47:16.753228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.753206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v7vms"]
Sep 10 02:47:16.753346 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.753241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrh84"]
Sep 10 02:47:16.753346 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.753266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9ngr6"]
Sep 10 02:47:16.753460 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.753379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:16.756384 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.756364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 02:47:16.756505 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.756435    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 02:47:16.756505 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.756463    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 02:47:16.756505 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.756476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5sdzz\""
Sep 10 02:47:16.769418 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.769401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:16.769503 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.769401    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9ngr6"]
Sep 10 02:47:16.772242 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.772224    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 02:47:16.772354 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.772229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gq58s\""
Sep 10 02:47:16.772354 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.772224    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 02:47:16.819563 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.819535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aefe13b-20ee-424f-8f13-225eee0253d5-data-volume\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.819563 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.819565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.819684 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.819612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27t2\" (UniqueName: \"kubernetes.io/projected/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-api-access-d27t2\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.819718 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.819678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aefe13b-20ee-424f-8f13-225eee0253d5-crio-socket\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.819718 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.819704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.920470 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f8gx\" (UniqueName: \"kubernetes.io/projected/6a65c613-798a-4bd4-bd72-e65189642bee-kube-api-access-6f8gx\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:16.920568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8lt\" (UniqueName: \"kubernetes.io/projected/55fccc6f-5bd7-47ad-9f36-5eeda877b4e4-kube-api-access-sj8lt\") pod \"ingress-canary-v7vms\" (UID: \"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4\") " pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:16.920568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aefe13b-20ee-424f-8f13-225eee0253d5-crio-socket\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.920568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.920568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a65c613-798a-4bd4-bd72-e65189642bee-config-volume\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:16.920568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aefe13b-20ee-424f-8f13-225eee0253d5-data-volume\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:16.920621    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:16.920673    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.420657323 +0000 UTC m=+34.570557028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aefe13b-20ee-424f-8f13-225eee0253d5-crio-socket\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a65c613-798a-4bd4-bd72-e65189642bee-tmp-dir\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:16.920703    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 02:47:16.920741 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27t2\" (UniqueName: \"kubernetes.io/projected/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-api-access-d27t2\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.920996 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:16.920769    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.42075323 +0000 UTC m=+34.570652929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : configmap "kube-rbac-proxy" not found
Sep 10 02:47:16.920996 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55fccc6f-5bd7-47ad-9f36-5eeda877b4e4-cert\") pod \"ingress-canary-v7vms\" (UID: \"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4\") " pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:16.920996 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.920911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aefe13b-20ee-424f-8f13-225eee0253d5-data-volume\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:16.930695 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:16.930671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27t2\" (UniqueName: \"kubernetes.io/projected/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-api-access-d27t2\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:17.021251 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:17.021251 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55fccc6f-5bd7-47ad-9f36-5eeda877b4e4-cert\") pod \"ingress-canary-v7vms\" (UID: \"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4\") " pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:17.021402 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f8gx\" (UniqueName: \"kubernetes.io/projected/6a65c613-798a-4bd4-bd72-e65189642bee-kube-api-access-6f8gx\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.021402 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8lt\" (UniqueName: \"kubernetes.io/projected/55fccc6f-5bd7-47ad-9f36-5eeda877b4e4-kube-api-access-sj8lt\") pod \"ingress-canary-v7vms\" (UID: \"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4\") " pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:17.021402 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a65c613-798a-4bd4-bd72-e65189642bee-config-volume\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.021402 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.021402 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.021343    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:17.021402 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.021396    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs podName:8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:49.021380409 +0000 UTC m=+66.171280101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs") pod "network-metrics-daemon-hs5b5" (UID: "8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:17.021691 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a65c613-798a-4bd4-bd72-e65189642bee-tmp-dir\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.021691 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.021427    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:17.021691 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.021509    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls podName:6a65c613-798a-4bd4-bd72-e65189642bee nodeName:}" failed. No retries permitted until 2026-09-10 02:47:17.521492789 +0000 UTC m=+34.671392479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls") pod "dns-default-9ngr6" (UID: "6a65c613-798a-4bd4-bd72-e65189642bee") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:17.021691 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a65c613-798a-4bd4-bd72-e65189642bee-tmp-dir\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.021820 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.021807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a65c613-798a-4bd4-bd72-e65189642bee-config-volume\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.023510 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.023489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55fccc6f-5bd7-47ad-9f36-5eeda877b4e4-cert\") pod \"ingress-canary-v7vms\" (UID: \"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4\") " pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:17.029648 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.029628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f8gx\" (UniqueName: \"kubernetes.io/projected/6a65c613-798a-4bd4-bd72-e65189642bee-kube-api-access-6f8gx\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.030284 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.030266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8lt\" (UniqueName: \"kubernetes.io/projected/55fccc6f-5bd7-47ad-9f36-5eeda877b4e4-kube-api-access-sj8lt\") pod \"ingress-canary-v7vms\" (UID: \"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4\") " pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:17.061754 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.061728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v7vms"
Sep 10 02:47:17.122042 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.122016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:17.122205 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.122186    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:17.122282 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.122271    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret podName:610b6131-5edf-4c48-a4d0-240d567c76c3 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:49.122250951 +0000 UTC m=+66.272150630 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret") pod "global-pull-secret-syncer-rjz9f" (UID: "610b6131-5edf-4c48-a4d0-240d567c76c3") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:17.223398 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.223372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:17.225680 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.223825    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:47:17.225680 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.223870    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:47:17.225680 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.223884    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hdnnx for pod openshift-network-diagnostics/network-check-target-z7xf8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:17.225680 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.223959    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx podName:1a798373-1bde-4671-83d5-0c1e6f678c33 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:49.223939786 +0000 UTC m=+66.373839478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdnnx" (UniqueName: "kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx") pod "network-check-target-z7xf8" (UID: "1a798373-1bde-4671-83d5-0c1e6f678c33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:17.371645 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.371617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v7vms"]
Sep 10 02:47:17.406969 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:17.406933    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55fccc6f_5bd7_47ad_9f36_5eeda877b4e4.slice/crio-3843e4f5de103ecbb1324b9b1622f2a1f02a43cd2a5a0fa54c913225da5fa6a3 WatchSource:0}: Error finding container 3843e4f5de103ecbb1324b9b1622f2a1f02a43cd2a5a0fa54c913225da5fa6a3: Status 404 returned error can't find the container with id 3843e4f5de103ecbb1324b9b1622f2a1f02a43cd2a5a0fa54c913225da5fa6a3
Sep 10 02:47:17.424933 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.424909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:17.425050 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.424959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:17.425109 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.425055    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 02:47:17.425109 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.425070    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:17.425262 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.425112    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.425097716 +0000 UTC m=+35.574997391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : configmap "kube-rbac-proxy" not found
Sep 10 02:47:17.425262 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.425127    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.425120978 +0000 UTC m=+35.575020654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:17.512543 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.512515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v7vms" event={"ID":"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4","Type":"ContainerStarted","Data":"3843e4f5de103ecbb1324b9b1622f2a1f02a43cd2a5a0fa54c913225da5fa6a3"}
Sep 10 02:47:17.515155 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.515132    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8ceac2a-9a49-4c9d-b922-d237e3940efa" containerID="6f2179f8d6dd1ea6f52b79a24ed37b23e43c1376c47996ff0d79fada9b85d988" exitCode=0
Sep 10 02:47:17.515258 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.515213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerDied","Data":"6f2179f8d6dd1ea6f52b79a24ed37b23e43c1376c47996ff0d79fada9b85d988"}
Sep 10 02:47:17.525275 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:17.525256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:17.525374 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.525363    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:17.525417 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:17.525412    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls podName:6a65c613-798a-4bd4-bd72-e65189642bee nodeName:}" failed. No retries permitted until 2026-09-10 02:47:18.52539506 +0000 UTC m=+35.675294737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls") pod "dns-default-9ngr6" (UID: "6a65c613-798a-4bd4-bd72-e65189642bee") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:18.143448 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.143236    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6vbtf_f2d9ef21-3103-4690-a438-6e727bd4b91e/dns-node-resolver/0.log"
Sep 10 02:47:18.319643 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.319438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:18.319643 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.319544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:18.319643 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.319552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:18.323702 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.323673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t98bg\""
Sep 10 02:47:18.323815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.323718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 02:47:18.323815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.323729    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 02:47:18.323815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.323768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 02:47:18.323815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.323795    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mtnpz\""
Sep 10 02:47:18.324075 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.324050    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 02:47:18.432456 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.432352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:18.432456 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.432410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:18.432873 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:18.432511    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 02:47:18.432873 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:18.432521    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:18.432873 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:18.432574    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:20.432558021 +0000 UTC m=+37.582457720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : configmap "kube-rbac-proxy" not found
Sep 10 02:47:18.432873 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:18.432592    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:20.432582673 +0000 UTC m=+37.582482351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:18.521248 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.521208    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8ceac2a-9a49-4c9d-b922-d237e3940efa" containerID="c05f6785c9ad1c9946698f74c81f727e6f4409613677eceabbf19f0da6a9d191" exitCode=0
Sep 10 02:47:18.521393 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.521254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerDied","Data":"c05f6785c9ad1c9946698f74c81f727e6f4409613677eceabbf19f0da6a9d191"}
Sep 10 02:47:18.533138 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:18.533113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:18.533296 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:18.533274    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:18.533376 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:18.533341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls podName:6a65c613-798a-4bd4-bd72-e65189642bee nodeName:}" failed. No retries permitted until 2026-09-10 02:47:20.533322125 +0000 UTC m=+37.683221806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls") pod "dns-default-9ngr6" (UID: "6a65c613-798a-4bd4-bd72-e65189642bee") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:19.137320 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:19.137290    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzhhr_fc590b11-0eeb-40df-885c-5f5f5b0dd195/node-ca/0.log"
Sep 10 02:47:19.527586 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:19.527508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg54z" event={"ID":"e8ceac2a-9a49-4c9d-b922-d237e3940efa","Type":"ContainerStarted","Data":"3a5ac0fff84ff83dd061320f0e47490c01739dcea8f46e570693f02b16e343cd"}
Sep 10 02:47:19.555912 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:19.555845    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wg54z" podStartSLOduration=8.163858149 podStartE2EDuration="36.555828065s" totalImagesPullingTime="28.391969916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:19.553939394 +0000 UTC m=+36.703839095" watchObservedRunningTime="2026-09-10 02:47:19.555828065 +0000 UTC m=+36.705727775"
Sep 10 02:47:20.449932 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:20.449891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:20.450097 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:20.449953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:20.450097 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:20.450056    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:20.450190 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:20.450118    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.450102875 +0000 UTC m=+41.600002555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:20.450190 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:20.450055    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 02:47:20.450272 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:20.450201    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.450190047 +0000 UTC m=+41.600089744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : configmap "kube-rbac-proxy" not found
Sep 10 02:47:20.531944 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:20.531913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v7vms" event={"ID":"55fccc6f-5bd7-47ad-9f36-5eeda877b4e4","Type":"ContainerStarted","Data":"54af492f30ad9157139a8796d697d2e42382a4a7f48c5d9a54dc42d7c39f1e59"}
Sep 10 02:47:20.550704 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:20.550682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:20.550815 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:20.550777    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:20.550854 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:20.550820    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls podName:6a65c613-798a-4bd4-bd72-e65189642bee nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.55080714 +0000 UTC m=+41.700706816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls") pod "dns-default-9ngr6" (UID: "6a65c613-798a-4bd4-bd72-e65189642bee") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:20.553433 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:20.553397    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v7vms" podStartSLOduration=2.093949201 podStartE2EDuration="4.553384194s" totalImagesPullingTime="2.459434993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:20.552357346 +0000 UTC m=+37.702257044" watchObservedRunningTime="2026-09-10 02:47:20.553384194 +0000 UTC m=+37.703283903"
Sep 10 02:47:21.733385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.733356    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"]
Sep 10 02:47:21.754434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.754405    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"]
Sep 10 02:47:21.754576 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.754506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.757238 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.757218    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 02:47:21.757438 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.757353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gk6mx\""
Sep 10 02:47:21.757490 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.757463    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 02:47:21.757591 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.757575    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 02:47:21.763358 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.763341    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 02:47:21.860474 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-trusted-ca\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.860583 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-bound-sa-token\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.860583 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-image-registry-private-configuration\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.860655 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sznm\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-kube-api-access-5sznm\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.860655 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/289759e2-95bd-4ca8-910b-e6d169232a1c-ca-trust-extracted\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.860655 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-installation-pull-secrets\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.860744 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.860779 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.860739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-certificates\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961216 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961216 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-certificates\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-trusted-ca\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-bound-sa-token\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-image-registry-private-configuration\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961385 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:21.961324    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 02:47:21.961385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sznm\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-kube-api-access-5sznm\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961385 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:21.961341    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb: secret "image-registry-tls" not found
Sep 10 02:47:21.961385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/289759e2-95bd-4ca8-910b-e6d169232a1c-ca-trust-extracted\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961717 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:21.961410    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls podName:289759e2-95bd-4ca8-910b-e6d169232a1c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:22.461391777 +0000 UTC m=+39.611291458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls") pod "image-registry-6dd7f7dbdd-9xjwb" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c") : secret "image-registry-tls" not found
Sep 10 02:47:21.961717 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-installation-pull-secrets\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961833 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/289759e2-95bd-4ca8-910b-e6d169232a1c-ca-trust-extracted\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.961960 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.961932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-certificates\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.962235 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.962218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-trusted-ca\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.964766 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.964738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-installation-pull-secrets\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.964766 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.964755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-image-registry-private-configuration\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.975532 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.975512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sznm\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-kube-api-access-5sznm\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:21.975717 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:21.975696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-bound-sa-token\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:22.465670 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.465453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:22.465834 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:22.465604    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 02:47:22.465834 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:22.465744    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb: secret "image-registry-tls" not found
Sep 10 02:47:22.465834 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:22.465797    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls podName:289759e2-95bd-4ca8-910b-e6d169232a1c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:23.465781173 +0000 UTC m=+40.615680869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls") pod "image-registry-6dd7f7dbdd-9xjwb" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c") : secret "image-registry-tls" not found
Sep 10 02:47:22.616549 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.616515    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"]
Sep 10 02:47:22.649450 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.649419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"]
Sep 10 02:47:22.649578 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.649510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"
Sep 10 02:47:22.652400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.652372    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 02:47:22.652517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.652379    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 02:47:22.653764 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.653749    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kgnjn\""
Sep 10 02:47:22.769036 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.768967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6krk\" (UniqueName: \"kubernetes.io/projected/e86d3abc-da18-4d1d-ae3f-bbe1335c9f10-kube-api-access-z6krk\") pod \"migrator-fdbb8fdff-mr7hd\" (UID: \"e86d3abc-da18-4d1d-ae3f-bbe1335c9f10\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"
Sep 10 02:47:22.870311 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.870280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6krk\" (UniqueName: \"kubernetes.io/projected/e86d3abc-da18-4d1d-ae3f-bbe1335c9f10-kube-api-access-z6krk\") pod \"migrator-fdbb8fdff-mr7hd\" (UID: \"e86d3abc-da18-4d1d-ae3f-bbe1335c9f10\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"
Sep 10 02:47:22.878911 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.878882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6krk\" (UniqueName: \"kubernetes.io/projected/e86d3abc-da18-4d1d-ae3f-bbe1335c9f10-kube-api-access-z6krk\") pod \"migrator-fdbb8fdff-mr7hd\" (UID: \"e86d3abc-da18-4d1d-ae3f-bbe1335c9f10\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"
Sep 10 02:47:22.958952 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:22.958929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"
Sep 10 02:47:23.244224 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.244114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd"]
Sep 10 02:47:23.246507 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:23.246478    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86d3abc_da18_4d1d_ae3f_bbe1335c9f10.slice/crio-47d17775a52cfec24ff790d64f3da28f9e138f16c8bb2b1215f7b62a6f739f44 WatchSource:0}: Error finding container 47d17775a52cfec24ff790d64f3da28f9e138f16c8bb2b1215f7b62a6f739f44: Status 404 returned error can't find the container with id 47d17775a52cfec24ff790d64f3da28f9e138f16c8bb2b1215f7b62a6f739f44
Sep 10 02:47:23.474599 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.474568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:23.474727 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:23.474710    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 02:47:23.474780 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:23.474729    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb: secret "image-registry-tls" not found
Sep 10 02:47:23.474780 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:23.474775    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls podName:289759e2-95bd-4ca8-910b-e6d169232a1c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:25.474761778 +0000 UTC m=+42.624661454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls") pod "image-registry-6dd7f7dbdd-9xjwb" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c") : secret "image-registry-tls" not found
Sep 10 02:47:23.538583 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.538548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd" event={"ID":"e86d3abc-da18-4d1d-ae3f-bbe1335c9f10","Type":"ContainerStarted","Data":"47d17775a52cfec24ff790d64f3da28f9e138f16c8bb2b1215f7b62a6f739f44"}
Sep 10 02:47:23.836878 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.836807    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-fqlrw"]
Sep 10 02:47:23.859525 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.859497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-fqlrw"]
Sep 10 02:47:23.859641 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.859581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:23.862651 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.862630    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 02:47:23.862797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.862697    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 02:47:23.862797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.862753    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 02:47:23.864033 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.864004    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 02:47:23.864226 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.864045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mpxwc\""
Sep 10 02:47:23.938715 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.938687    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-4sjnx"]
Sep 10 02:47:23.953779 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.953757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-4sjnx"]
Sep 10 02:47:23.953914 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.953852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx"
Sep 10 02:47:23.956783 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.956761    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zbcc5\""
Sep 10 02:47:23.977864 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.977842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7102830-b623-45ce-84a8-40cddfb3f289-signing-key\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:23.977985 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.977900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htd6c\" (UniqueName: \"kubernetes.io/projected/c7102830-b623-45ce-84a8-40cddfb3f289-kube-api-access-htd6c\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:23.978045 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:23.977984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7102830-b623-45ce-84a8-40cddfb3f289-signing-cabundle\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.079219 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.079186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htd6c\" (UniqueName: \"kubernetes.io/projected/c7102830-b623-45ce-84a8-40cddfb3f289-kube-api-access-htd6c\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.079378 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.079260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7102830-b623-45ce-84a8-40cddfb3f289-signing-cabundle\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.079378 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.079348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7102830-b623-45ce-84a8-40cddfb3f289-signing-key\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.079378 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.079374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkstk\" (UniqueName: \"kubernetes.io/projected/864fa180-f881-4d9f-89ff-6d98ab727c3d-kube-api-access-tkstk\") pod \"network-check-source-685456655f-4sjnx\" (UID: \"864fa180-f881-4d9f-89ff-6d98ab727c3d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx"
Sep 10 02:47:24.080013 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.079983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7102830-b623-45ce-84a8-40cddfb3f289-signing-cabundle\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.082220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.082198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7102830-b623-45ce-84a8-40cddfb3f289-signing-key\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.088238 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.088176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htd6c\" (UniqueName: \"kubernetes.io/projected/c7102830-b623-45ce-84a8-40cddfb3f289-kube-api-access-htd6c\") pod \"service-ca-75d99c484-fqlrw\" (UID: \"c7102830-b623-45ce-84a8-40cddfb3f289\") " pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.167956 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.167926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-fqlrw"
Sep 10 02:47:24.180288 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.180268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkstk\" (UniqueName: \"kubernetes.io/projected/864fa180-f881-4d9f-89ff-6d98ab727c3d-kube-api-access-tkstk\") pod \"network-check-source-685456655f-4sjnx\" (UID: \"864fa180-f881-4d9f-89ff-6d98ab727c3d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx"
Sep 10 02:47:24.188479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.188456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkstk\" (UniqueName: \"kubernetes.io/projected/864fa180-f881-4d9f-89ff-6d98ab727c3d-kube-api-access-tkstk\") pod \"network-check-source-685456655f-4sjnx\" (UID: \"864fa180-f881-4d9f-89ff-6d98ab727c3d\") " pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx"
Sep 10 02:47:24.264699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.264673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx"
Sep 10 02:47:24.465018 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.464987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-fqlrw"]
Sep 10 02:47:24.482938 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.482915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:24.483056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.482956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:24.483056 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:24.483043    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 02:47:24.483182 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:24.483070    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:24.483182 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:24.483110    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:32.483090013 +0000 UTC m=+49.632989707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : configmap "kube-rbac-proxy" not found
Sep 10 02:47:24.483182 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:24.483125    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:32.483118171 +0000 UTC m=+49.633017848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:24.560559 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.560371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-4sjnx"]
Sep 10 02:47:24.583798 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:24.583770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:24.583950 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:24.583937    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:24.584010 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:24.583998    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls podName:6a65c613-798a-4bd4-bd72-e65189642bee nodeName:}" failed. No retries permitted until 2026-09-10 02:47:32.583979989 +0000 UTC m=+49.733879667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls") pod "dns-default-9ngr6" (UID: "6a65c613-798a-4bd4-bd72-e65189642bee") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:24.680341 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:24.680311    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7102830_b623_45ce_84a8_40cddfb3f289.slice/crio-6ec6307e1526e04c2479c76c2b6f89b617ca7b4765b17afa769ad70a2b28dda7 WatchSource:0}: Error finding container 6ec6307e1526e04c2479c76c2b6f89b617ca7b4765b17afa769ad70a2b28dda7: Status 404 returned error can't find the container with id 6ec6307e1526e04c2479c76c2b6f89b617ca7b4765b17afa769ad70a2b28dda7
Sep 10 02:47:24.680753 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:24.680728    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864fa180_f881_4d9f_89ff_6d98ab727c3d.slice/crio-149ef14e35cff50695293bc993ccb913d8f548f1118e22acfd07c19efc3f93b2 WatchSource:0}: Error finding container 149ef14e35cff50695293bc993ccb913d8f548f1118e22acfd07c19efc3f93b2: Status 404 returned error can't find the container with id 149ef14e35cff50695293bc993ccb913d8f548f1118e22acfd07c19efc3f93b2
Sep 10 02:47:25.492016 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:25.491971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:25.492449 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:25.492155    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 02:47:25.492449 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:25.492197    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb: secret "image-registry-tls" not found
Sep 10 02:47:25.492449 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:25.492267    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls podName:289759e2-95bd-4ca8-910b-e6d169232a1c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:29.492245196 +0000 UTC m=+46.642144886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls") pod "image-registry-6dd7f7dbdd-9xjwb" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c") : secret "image-registry-tls" not found
Sep 10 02:47:25.545817 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:25.545784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-fqlrw" event={"ID":"c7102830-b623-45ce-84a8-40cddfb3f289","Type":"ContainerStarted","Data":"6ec6307e1526e04c2479c76c2b6f89b617ca7b4765b17afa769ad70a2b28dda7"}
Sep 10 02:47:25.547855 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:25.547819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd" event={"ID":"e86d3abc-da18-4d1d-ae3f-bbe1335c9f10","Type":"ContainerStarted","Data":"702301441ca2df747d0dae113cad5cf1eca01a430f8f4911ea04977629ec37d2"}
Sep 10 02:47:25.547971 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:25.547863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd" event={"ID":"e86d3abc-da18-4d1d-ae3f-bbe1335c9f10","Type":"ContainerStarted","Data":"b87a4a6939fdb31ddd12748496cffa69b1ffb2e752b468eb835496873dde042e"}
Sep 10 02:47:25.549128 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:25.549098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx" event={"ID":"864fa180-f881-4d9f-89ff-6d98ab727c3d","Type":"ContainerStarted","Data":"149ef14e35cff50695293bc993ccb913d8f548f1118e22acfd07c19efc3f93b2"}
Sep 10 02:47:25.569889 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:25.569843    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mr7hd" podStartSLOduration=1.788756037 podStartE2EDuration="3.569831172s" totalImagesPullingTime="1.781075135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:25.568232279 +0000 UTC m=+42.718131978" watchObservedRunningTime="2026-09-10 02:47:25.569831172 +0000 UTC m=+42.719730869"
Sep 10 02:47:27.557038 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:27.556960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-fqlrw" event={"ID":"c7102830-b623-45ce-84a8-40cddfb3f289","Type":"ContainerStarted","Data":"1f5e549376128a282be83e38b8995377f5123d55c1ea6074799b7351be1f9e47"}
Sep 10 02:47:27.575820 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:27.575776    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-fqlrw" podStartSLOduration=2.053920398 podStartE2EDuration="4.575756286s" totalImagesPullingTime="2.521835888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:27.574726544 +0000 UTC m=+44.724626244" watchObservedRunningTime="2026-09-10 02:47:27.575756286 +0000 UTC m=+44.725655987"
Sep 10 02:47:29.523744 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:29.523713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:29.524233 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:29.523890    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 02:47:29.524233 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:29.523912    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb: secret "image-registry-tls" not found
Sep 10 02:47:29.524233 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:29.523976    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls podName:289759e2-95bd-4ca8-910b-e6d169232a1c nodeName:}" failed. No retries permitted until 2026-09-10 02:47:37.523956697 +0000 UTC m=+54.673856377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls") pod "image-registry-6dd7f7dbdd-9xjwb" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c") : secret "image-registry-tls" not found
Sep 10 02:47:29.562945 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:29.562913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx" event={"ID":"864fa180-f881-4d9f-89ff-6d98ab727c3d","Type":"ContainerStarted","Data":"e3acf73e991e9b0c51ed288c6e6442a73b00e435a823005479271fd03b241d44"}
Sep 10 02:47:29.577970 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:29.577921    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-4sjnx" podStartSLOduration=2.35241852 podStartE2EDuration="6.577905337s" totalImagesPullingTime="4.225486817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:29.577806071 +0000 UTC m=+46.727705769" watchObservedRunningTime="2026-09-10 02:47:29.577905337 +0000 UTC m=+46.727805237"
Sep 10 02:47:32.544470 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:32.544432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:32.544920 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:32.544550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:32.544920 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:32.544651    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:32.544920 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:32.544719    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls podName:5aefe13b-20ee-424f-8f13-225eee0253d5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:48.544700716 +0000 UTC m=+65.694600414 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrh84" (UID: "5aefe13b-20ee-424f-8f13-225eee0253d5") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:32.545082 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:32.545030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aefe13b-20ee-424f-8f13-225eee0253d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:32.645638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:32.645609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:32.647789 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:32.647771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a65c613-798a-4bd4-bd72-e65189642bee-metrics-tls\") pod \"dns-default-9ngr6\" (UID: \"6a65c613-798a-4bd4-bd72-e65189642bee\") " pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:32.677003 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:32.676974    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:32.972269 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:32.972239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9ngr6"]
Sep 10 02:47:32.976120 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:32.976091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a65c613_798a_4bd4_bd72_e65189642bee.slice/crio-67b70503fd2fafe968560afcfbd31b94b42e2f827bc8408fcebc03c5d4a5b0b9 WatchSource:0}: Error finding container 67b70503fd2fafe968560afcfbd31b94b42e2f827bc8408fcebc03c5d4a5b0b9: Status 404 returned error can't find the container with id 67b70503fd2fafe968560afcfbd31b94b42e2f827bc8408fcebc03c5d4a5b0b9
Sep 10 02:47:33.574644 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:33.574609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9ngr6" event={"ID":"6a65c613-798a-4bd4-bd72-e65189642bee","Type":"ContainerStarted","Data":"67b70503fd2fafe968560afcfbd31b94b42e2f827bc8408fcebc03c5d4a5b0b9"}
Sep 10 02:47:35.583354 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:35.583316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9ngr6" event={"ID":"6a65c613-798a-4bd4-bd72-e65189642bee","Type":"ContainerStarted","Data":"e84f13017aaacec8c56a0618a0da40d99fa38fb3f4db8cc341892c651033a3c9"}
Sep 10 02:47:35.583714 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:35.583409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9ngr6" event={"ID":"6a65c613-798a-4bd4-bd72-e65189642bee","Type":"ContainerStarted","Data":"0fd44427cd15cbbdf952eb29a9596ff548b74ac906cb7daf96b846af07b76eb3"}
Sep 10 02:47:35.583714 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:35.583438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:35.604803 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:35.604760    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9ngr6" podStartSLOduration=17.831928156 podStartE2EDuration="19.604744944s" totalImagesPullingTime="1.772816788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:35.604734147 +0000 UTC m=+52.754633856" watchObservedRunningTime="2026-09-10 02:47:35.604744944 +0000 UTC m=+52.754644636"
Sep 10 02:47:37.581651 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:37.581610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:37.584063 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:37.584037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"image-registry-6dd7f7dbdd-9xjwb\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") " pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:37.663913 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:37.663892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:37.972375 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:37.972344    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"]
Sep 10 02:47:37.978878 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:37.978841    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289759e2_95bd_4ca8_910b_e6d169232a1c.slice/crio-8e7a4a14860ae6b740bf1a55db43a979c1a0257dc207ba34bca3010e541e867d WatchSource:0}: Error finding container 8e7a4a14860ae6b740bf1a55db43a979c1a0257dc207ba34bca3010e541e867d: Status 404 returned error can't find the container with id 8e7a4a14860ae6b740bf1a55db43a979c1a0257dc207ba34bca3010e541e867d
Sep 10 02:47:38.595628 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:38.595593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb" event={"ID":"289759e2-95bd-4ca8-910b-e6d169232a1c","Type":"ContainerStarted","Data":"f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d"}
Sep 10 02:47:38.596091 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:38.595645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb" event={"ID":"289759e2-95bd-4ca8-910b-e6d169232a1c","Type":"ContainerStarted","Data":"8e7a4a14860ae6b740bf1a55db43a979c1a0257dc207ba34bca3010e541e867d"}
Sep 10 02:47:38.596091 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:38.595733    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:38.617225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:38.617182    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb" podStartSLOduration=17.617150379999998 podStartE2EDuration="17.61715038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:38.616623742 +0000 UTC m=+55.766523439" watchObservedRunningTime="2026-09-10 02:47:38.61715038 +0000 UTC m=+55.767050078"
Sep 10 02:47:42.509315 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:42.509285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-72mqv"
Sep 10 02:47:43.713736 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.713701    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"]
Sep 10 02:47:43.763388 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.763358    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-556d97688d-77wr5"]
Sep 10 02:47:43.766278 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.766263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.790221 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.790198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556d97688d-77wr5"]
Sep 10 02:47:43.818699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-registry-certificates\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.818699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-installation-pull-secrets\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.818838 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-registry-tls\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.818838 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-ca-trust-extracted\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.818838 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-trusted-ca\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.818940 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-bound-sa-token\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.818940 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-image-registry-private-configuration\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.818940 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.818915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j96jb\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-kube-api-access-j96jb\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.919944 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.919918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-ca-trust-extracted\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920049 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.919949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-trusted-ca\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920049 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.919983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-bound-sa-token\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920049 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.920007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-image-registry-private-configuration\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920049 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.920038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j96jb\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-kube-api-access-j96jb\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920271 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.920110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-registry-certificates\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920271 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.920135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-installation-pull-secrets\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920388 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.920199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-registry-tls\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920441 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.920400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-ca-trust-extracted\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.920933 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.920904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-registry-certificates\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.921124 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.921099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-trusted-ca\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.922661 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.922635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-image-registry-private-configuration\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.922727 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.922642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-installation-pull-secrets\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.922790 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.922770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-registry-tls\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.928765 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.928738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-bound-sa-token\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:43.929842 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:43.929823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j96jb\" (UniqueName: \"kubernetes.io/projected/9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305-kube-api-access-j96jb\") pod \"image-registry-556d97688d-77wr5\" (UID: \"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305\") " pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:44.074496 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:44.074431    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:44.345418 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:44.345388    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556d97688d-77wr5"]
Sep 10 02:47:44.347993 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:44.347966    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9010e46f_c5b2_4f62_8bf1_d6d6ddd6b305.slice/crio-495b629695488fc0a85ad560c962303535cce24aabc7bc13b0f36b298767b425 WatchSource:0}: Error finding container 495b629695488fc0a85ad560c962303535cce24aabc7bc13b0f36b298767b425: Status 404 returned error can't find the container with id 495b629695488fc0a85ad560c962303535cce24aabc7bc13b0f36b298767b425
Sep 10 02:47:44.618731 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:44.618645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556d97688d-77wr5" event={"ID":"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305","Type":"ContainerStarted","Data":"f5c4ea55057789d13218354dcd90ad6b73438108467208b0f488f7bda2240b1b"}
Sep 10 02:47:44.618731 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:44.618727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556d97688d-77wr5" event={"ID":"9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305","Type":"ContainerStarted","Data":"495b629695488fc0a85ad560c962303535cce24aabc7bc13b0f36b298767b425"}
Sep 10 02:47:44.618955 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:44.618778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:47:44.637684 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:44.637635    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-556d97688d-77wr5" podStartSLOduration=1.637621737 podStartE2EDuration="1.637621737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:44.637040033 +0000 UTC m=+61.786939731" watchObservedRunningTime="2026-09-10 02:47:44.637621737 +0000 UTC m=+61.787521435"
Sep 10 02:47:45.588067 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:45.588040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9ngr6"
Sep 10 02:47:48.551207 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:48.551149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:48.553413 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:48.553390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aefe13b-20ee-424f-8f13-225eee0253d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrh84\" (UID: \"5aefe13b-20ee-424f-8f13-225eee0253d5\") " pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:48.852261 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:48.852237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fq77l\""
Sep 10 02:47:48.860692 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:48.860668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrh84"
Sep 10 02:47:49.047645 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.047615    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"]
Sep 10 02:47:49.050321 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.050303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"
Sep 10 02:47:49.055009 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.054985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 02:47:49.055282 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.055263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v8jl6\""
Sep 10 02:47:49.055398 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.055381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:49.058067 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.058050    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 02:47:49.065311 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.065292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"]
Sep 10 02:47:49.068460 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.068436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5-metrics-certs\") pod \"network-metrics-daemon-hs5b5\" (UID: \"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5\") " pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:49.132600 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.132545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrh84"]
Sep 10 02:47:49.135851 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:49.135823    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aefe13b_20ee_424f_8f13_225eee0253d5.slice/crio-9c3a1c7c6cda382881e5f53f1818432329686f8c7b8b55f8ff53d464a91ad66a WatchSource:0}: Error finding container 9c3a1c7c6cda382881e5f53f1818432329686f8c7b8b55f8ff53d464a91ad66a: Status 404 returned error can't find the container with id 9c3a1c7c6cda382881e5f53f1818432329686f8c7b8b55f8ff53d464a91ad66a
Sep 10 02:47:49.156687 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.156662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-294n7\" (UID: \"5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"
Sep 10 02:47:49.156770 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.156722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:49.159745 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.159723    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 02:47:49.168868 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.168845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610b6131-5edf-4c48-a4d0-240d567c76c3-original-pull-secret\") pod \"global-pull-secret-syncer-rjz9f\" (UID: \"610b6131-5edf-4c48-a4d0-240d567c76c3\") " pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:49.239074 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.239032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjz9f"
Sep 10 02:47:49.248415 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.248217    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mtnpz\""
Sep 10 02:47:49.255535 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.255514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hs5b5"
Sep 10 02:47:49.257368 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.257347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:49.257452 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.257416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-294n7\" (UID: \"5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"
Sep 10 02:47:49.260213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.259952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-294n7\" (UID: \"5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"
Sep 10 02:47:49.260354 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.260285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnnx\" (UniqueName: \"kubernetes.io/projected/1a798373-1bde-4671-83d5-0c1e6f678c33-kube-api-access-hdnnx\") pod \"network-check-target-z7xf8\" (UID: \"1a798373-1bde-4671-83d5-0c1e6f678c33\") " pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:49.359218 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.359157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"
Sep 10 02:47:49.518460 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.518430    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rjz9f"]
Sep 10 02:47:49.521277 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:49.521253    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610b6131_5edf_4c48_a4d0_240d567c76c3.slice/crio-8e21be954d7d4e1485f477645e53aebe79fb9edd542f73618c415d558c8e57b9 WatchSource:0}: Error finding container 8e21be954d7d4e1485f477645e53aebe79fb9edd542f73618c415d558c8e57b9: Status 404 returned error can't find the container with id 8e21be954d7d4e1485f477645e53aebe79fb9edd542f73618c415d558c8e57b9
Sep 10 02:47:49.540403 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.540381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hs5b5"]
Sep 10 02:47:49.542450 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.542433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t98bg\""
Sep 10 02:47:49.543035 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:49.543011    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f51e98a_41a2_4eaa_b6c4_0f106d13d4f5.slice/crio-d2290238a4094019189cce87dfd3f24998b7963e2ed8cb9edf6744e4aae29543 WatchSource:0}: Error finding container d2290238a4094019189cce87dfd3f24998b7963e2ed8cb9edf6744e4aae29543: Status 404 returned error can't find the container with id d2290238a4094019189cce87dfd3f24998b7963e2ed8cb9edf6744e4aae29543
Sep 10 02:47:49.552385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.552367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:49.637285 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.637249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hs5b5" event={"ID":"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5","Type":"ContainerStarted","Data":"d2290238a4094019189cce87dfd3f24998b7963e2ed8cb9edf6744e4aae29543"}
Sep 10 02:47:49.638448 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.638408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rjz9f" event={"ID":"610b6131-5edf-4c48-a4d0-240d567c76c3","Type":"ContainerStarted","Data":"8e21be954d7d4e1485f477645e53aebe79fb9edd542f73618c415d558c8e57b9"}
Sep 10 02:47:49.639910 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.639884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"]
Sep 10 02:47:49.640652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.640626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrh84" event={"ID":"5aefe13b-20ee-424f-8f13-225eee0253d5","Type":"ContainerStarted","Data":"5b14c10417921dcc3a9165891351cbd4c9fe3c1423a3bcec14f83994be7b3238"}
Sep 10 02:47:49.640748 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.640684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrh84" event={"ID":"5aefe13b-20ee-424f-8f13-225eee0253d5","Type":"ContainerStarted","Data":"9c3a1c7c6cda382881e5f53f1818432329686f8c7b8b55f8ff53d464a91ad66a"}
Sep 10 02:47:49.643527 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:49.643501    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2aa94b_ba0c_49d6_8225_ec6daaf68f9d.slice/crio-cfd15de03a87c648d212782b2d762a9ac632ef5e4e1fe9730e938750c8dccfd2 WatchSource:0}: Error finding container cfd15de03a87c648d212782b2d762a9ac632ef5e4e1fe9730e938750c8dccfd2: Status 404 returned error can't find the container with id cfd15de03a87c648d212782b2d762a9ac632ef5e4e1fe9730e938750c8dccfd2
Sep 10 02:47:49.838069 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:49.837992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7xf8"]
Sep 10 02:47:49.865115 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:49.865081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a798373_1bde_4671_83d5_0c1e6f678c33.slice/crio-8315cd55ee254845bff8bfc241a7f507026eeea2ea84fab52724b5e8d1da7aa2 WatchSource:0}: Error finding container 8315cd55ee254845bff8bfc241a7f507026eeea2ea84fab52724b5e8d1da7aa2: Status 404 returned error can't find the container with id 8315cd55ee254845bff8bfc241a7f507026eeea2ea84fab52724b5e8d1da7aa2
Sep 10 02:47:50.647051 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:50.647010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrh84" event={"ID":"5aefe13b-20ee-424f-8f13-225eee0253d5","Type":"ContainerStarted","Data":"cbf9ba9ecc51ab29544a636778d15e1e47584f4b5b1633336aa6588f033d55e3"}
Sep 10 02:47:50.648797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:50.648766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7xf8" event={"ID":"1a798373-1bde-4671-83d5-0c1e6f678c33","Type":"ContainerStarted","Data":"d90481153101ef6a51bfcf3b8d7203ff411ab658d886bb43b8fbd13ec8af6887"}
Sep 10 02:47:50.648935 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:50.648817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7xf8" event={"ID":"1a798373-1bde-4671-83d5-0c1e6f678c33","Type":"ContainerStarted","Data":"8315cd55ee254845bff8bfc241a7f507026eeea2ea84fab52724b5e8d1da7aa2"}
Sep 10 02:47:50.648935 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:50.648859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:47:50.650372 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:50.650345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7" event={"ID":"5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d","Type":"ContainerStarted","Data":"cfd15de03a87c648d212782b2d762a9ac632ef5e4e1fe9730e938750c8dccfd2"}
Sep 10 02:47:50.670222 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:50.670173    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z7xf8" podStartSLOduration=67.670146981 podStartE2EDuration="1m7.670146981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:50.668128102 +0000 UTC m=+67.818027801" watchObservedRunningTime="2026-09-10 02:47:50.670146981 +0000 UTC m=+67.820046742"
Sep 10 02:47:51.656199 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:51.656138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hs5b5" event={"ID":"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5","Type":"ContainerStarted","Data":"fbd8a508c0787ab60feff010006a82b2dee705313ddb0c2c138b1ca4d1f08693"}
Sep 10 02:47:51.656635 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:51.656231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hs5b5" event={"ID":"8f51e98a-41a2-4eaa-b6c4-0f106d13d4f5","Type":"ContainerStarted","Data":"f1e20d5d38279394ce55d32f7677e3b2b11c7ff2cb64014c86270541f14ef292"}
Sep 10 02:47:51.657970 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:51.657929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7" event={"ID":"5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d","Type":"ContainerStarted","Data":"ea2c92641db90bd53af4429ce0c73bdbacf39df453e99974626b2af0d8559ce3"}
Sep 10 02:47:51.658104 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:51.658000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"
Sep 10 02:47:51.663617 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:51.663590    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7"
Sep 10 02:47:51.675952 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:51.675905    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hs5b5" podStartSLOduration=67.2976827 podStartE2EDuration="1m8.67589101s" totalImagesPullingTime="1.37820831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:51.675281807 +0000 UTC m=+68.825181507" watchObservedRunningTime="2026-09-10 02:47:51.67589101 +0000 UTC m=+68.825790708"
Sep 10 02:47:51.695399 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:51.695358    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-294n7" podStartSLOduration=1.420875225 podStartE2EDuration="2.695345068s" totalImagesPullingTime="1.274469843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:51.694243676 +0000 UTC m=+68.844143374" watchObservedRunningTime="2026-09-10 02:47:51.695345068 +0000 UTC m=+68.845244763"
Sep 10 02:47:52.664350 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:52.664314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrh84" event={"ID":"5aefe13b-20ee-424f-8f13-225eee0253d5","Type":"ContainerStarted","Data":"15af9f7132e8af59208ffcd220ed294bbea12f47253f726f220ab62437d49648"}
Sep 10 02:47:52.681723 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:52.681504    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lrh84" podStartSLOduration=33.938810657 podStartE2EDuration="36.681483764s" totalImagesPullingTime="2.742673107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:47:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:52.680863923 +0000 UTC m=+69.830763622" watchObservedRunningTime="2026-09-10 02:47:52.681483764 +0000 UTC m=+69.831383463"
Sep 10 02:47:53.721284 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:53.721251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:47:54.673084 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:54.673047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rjz9f" event={"ID":"610b6131-5edf-4c48-a4d0-240d567c76c3","Type":"ContainerStarted","Data":"c4b0e46ee0486bf2bddb4f6dd9aca8c64eeb3d4cddd0016dd9e0dcc2f4b167db"}
Sep 10 02:47:57.452615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.452551    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rjz9f" podStartSLOduration=70.065238277 podStartE2EDuration="1m14.452527351s" totalImagesPullingTime="4.387289074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:54.689960947 +0000 UTC m=+71.839860644" watchObservedRunningTime="2026-09-10 02:47:57.452527351 +0000 UTC m=+74.602427052"
Sep 10 02:47:57.453647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.453619    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"]
Sep 10 02:47:57.457348 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.457329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.463182 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.460395    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 02:47:57.463182 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.460677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 02:47:57.463182 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.462296    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9q9bz\""
Sep 10 02:47:57.463182 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.462338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 02:47:57.464867 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.463941    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 02:47:57.464867 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.464203    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 02:47:57.466632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.466612    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"]
Sep 10 02:47:57.469634 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.469616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.473009 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.472990    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"]
Sep 10 02:47:57.474222 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.474206    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 02:47:57.474360 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.474340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2dt4q\""
Sep 10 02:47:57.474429 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.474406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 02:47:57.474512 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.474232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 02:47:57.490131 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.487502    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"]
Sep 10 02:47:57.491431 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.491374    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9svq4"]
Sep 10 02:47:57.494772 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.494755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.497893 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.497687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 02:47:57.497893 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.497764    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 02:47:57.497893 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.497781    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9ltsg\""
Sep 10 02:47:57.498093 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.498075    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 02:47:57.512908 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.512887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.512986 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.512927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.512986 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.512961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-root\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.513132 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.512999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a23e89-45ac-4a22-a437-5da898f454d6-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.513132 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.513031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-tls\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.513132 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.513084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-metrics-client-ca\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.513132 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.513106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv278\" (UniqueName: \"kubernetes.io/projected/52a23e89-45ac-4a22-a437-5da898f454d6-kube-api-access-cv278\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.513352 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.513142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.513352 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.513194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/52a23e89-45ac-4a22-a437-5da898f454d6-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.513352 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.513224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-sys\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.513352 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.513260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-textfile\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.514053 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.514389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-accelerators-collector-config\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.514389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5hb\" (UniqueName: \"kubernetes.io/projected/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-kube-api-access-7w5hb\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.514389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.514389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.514389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-wtmp\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.514389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrlf\" (UniqueName: \"kubernetes.io/projected/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-kube-api-access-ktrlf\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.514389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.514269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.615157 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.615301 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.615301 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-root\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615301 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a23e89-45ac-4a22-a437-5da898f454d6-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.615301 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-tls\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-metrics-client-ca\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv278\" (UniqueName: \"kubernetes.io/projected/52a23e89-45ac-4a22-a437-5da898f454d6-kube-api-access-cv278\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.615489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/52a23e89-45ac-4a22-a437-5da898f454d6-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.615489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-sys\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-textfile\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-accelerators-collector-config\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5hb\" (UniqueName: \"kubernetes.io/projected/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-kube-api-access-7w5hb\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-wtmp\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrlf\" (UniqueName: \"kubernetes.io/projected/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-kube-api-access-ktrlf\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.615830 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:57.615780    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 02:47:57.616219 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:57.615843    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-tls podName:9c53091c-e6fc-48c8-a734-bc27cffe7c7a nodeName:}" failed. No retries permitted until 2026-09-10 02:47:58.115824671 +0000 UTC m=+75.265724352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-s77n5" (UID: "9c53091c-e6fc-48c8-a734-bc27cffe7c7a") : secret "openshift-state-metrics-tls" not found
Sep 10 02:47:57.616219 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.616219 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.615911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-sys\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.616371 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.616335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-textfile\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.616524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-root\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.616559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.616831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/52a23e89-45ac-4a22-a437-5da898f454d6-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.616848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-wtmp\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:57.617064    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:57.617109    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-tls podName:52a23e89-45ac-4a22-a437-5da898f454d6 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:58.117094297 +0000 UTC m=+75.266993978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-6fmh5" (UID: "52a23e89-45ac-4a22-a437-5da898f454d6") : secret "kube-state-metrics-tls" not found
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:57.617195    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:47:57.617227    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-tls podName:5e5bb6c7-94e4-4ea5-bb1f-34c41b535988 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:58.117216185 +0000 UTC m=+75.267115867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-tls") pod "node-exporter-9svq4" (UID: "5e5bb6c7-94e4-4ea5-bb1f-34c41b535988") : secret "node-exporter-tls" not found
Sep 10 02:47:57.617367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.617329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-accelerators-collector-config\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.617849 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.617598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a23e89-45ac-4a22-a437-5da898f454d6-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.617849 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.617842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-metrics-client-ca\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.619941 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.619916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.619941 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.620040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.619941 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.620521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:57.627600 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.627555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrlf\" (UniqueName: \"kubernetes.io/projected/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-kube-api-access-ktrlf\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:57.628440 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.628399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv278\" (UniqueName: \"kubernetes.io/projected/52a23e89-45ac-4a22-a437-5da898f454d6-kube-api-access-cv278\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:57.629384 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:57.629360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5hb\" (UniqueName: \"kubernetes.io/projected/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-kube-api-access-7w5hb\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:58.119226 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.119193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-tls\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:58.119391 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.119248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:58.119391 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.119269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:58.121607 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.121573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e5bb6c7-94e4-4ea5-bb1f-34c41b535988-node-exporter-tls\") pod \"node-exporter-9svq4\" (UID: \"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988\") " pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:58.121724 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.121642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c53091c-e6fc-48c8-a734-bc27cffe7c7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-s77n5\" (UID: \"9c53091c-e6fc-48c8-a734-bc27cffe7c7a\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:58.121724 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.121698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/52a23e89-45ac-4a22-a437-5da898f454d6-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6fmh5\" (UID: \"52a23e89-45ac-4a22-a437-5da898f454d6\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:58.374528 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.374448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"
Sep 10 02:47:58.381775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.381749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"
Sep 10 02:47:58.404657 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.404625    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9svq4"
Sep 10 02:47:58.517111 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.517046    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:47:58.521915 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.521893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.524674 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.524650    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 02:47:58.524674 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.524652    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 02:47:58.524850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.524723    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 02:47:58.524850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.524774    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 02:47:58.525083 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.525062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 02:47:58.525199 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.525095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 02:47:58.525199 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.525100    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 02:47:58.525199 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.525068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 02:47:58.525199 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.525147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dlmdm\""
Sep 10 02:47:58.525419 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.525405    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 02:47:58.534447 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.534429    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:47:58.622329 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622464 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622464 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622464 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-volume\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622464 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622464 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622656 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622656 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-out\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622656 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622656 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-web-config\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.622797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.622723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbd7\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-kube-api-access-wzbd7\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.654230 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.654144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5"]
Sep 10 02:47:58.657209 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:58.657184    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c53091c_e6fc_48c8_a734_bc27cffe7c7a.slice/crio-05abf9e73cd258a1f425fd33b4ed392d2e603eac64eb1158414f5e6485fe509d WatchSource:0}: Error finding container 05abf9e73cd258a1f425fd33b4ed392d2e603eac64eb1158414f5e6485fe509d: Status 404 returned error can't find the container with id 05abf9e73cd258a1f425fd33b4ed392d2e603eac64eb1158414f5e6485fe509d
Sep 10 02:47:58.662615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.662592    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5"]
Sep 10 02:47:58.666510 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:58.666484    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a23e89_45ac_4a22_a437_5da898f454d6.slice/crio-d15662f3601130380ee720eeddc95065a3cd55237de21fbab32280fbda620616 WatchSource:0}: Error finding container d15662f3601130380ee720eeddc95065a3cd55237de21fbab32280fbda620616: Status 404 returned error can't find the container with id d15662f3601130380ee720eeddc95065a3cd55237de21fbab32280fbda620616
Sep 10 02:47:58.689631 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.689605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9svq4" event={"ID":"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988","Type":"ContainerStarted","Data":"833a105e18ed41f14e7e4aec0e19e237b11596b9ab703eab822c7d478aabc8ba"}
Sep 10 02:47:58.690734 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.690714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5" event={"ID":"52a23e89-45ac-4a22-a437-5da898f454d6","Type":"ContainerStarted","Data":"d15662f3601130380ee720eeddc95065a3cd55237de21fbab32280fbda620616"}
Sep 10 02:47:58.691789 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.691768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5" event={"ID":"9c53091c-e6fc-48c8-a734-bc27cffe7c7a","Type":"ContainerStarted","Data":"05abf9e73cd258a1f425fd33b4ed392d2e603eac64eb1158414f5e6485fe509d"}
Sep 10 02:47:58.723957 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.723936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbd7\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-kube-api-access-wzbd7\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724094 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.723974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724094 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724094 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724094 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-volume\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724094 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724356 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724356 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724356 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-out\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724356 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724356 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724585 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-web-config\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.724585 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.724404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.725109 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.725084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.725545 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.725516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.725655 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.725533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.727507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.727461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-out\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.728434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.727954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.728434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.728030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.728434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.728395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.728631 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.728520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-web-config\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.728814 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.728791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.730133 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.730108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.730644 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.730620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-volume\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.730806 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.730784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.737131 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.737108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbd7\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-kube-api-access-wzbd7\") pod \"alertmanager-main-0\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:58.831941 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:58.831905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:47:59.119390 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:59.119348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:47:59.223744 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:47:59.223713    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb3a3be_3403_401a_a89c_a71103ba9cc4.slice/crio-7f778e16dd86528c0d721dd474dde58ddc7394ff01d1b255aa758a5937aec74a WatchSource:0}: Error finding container 7f778e16dd86528c0d721dd474dde58ddc7394ff01d1b255aa758a5937aec74a: Status 404 returned error can't find the container with id 7f778e16dd86528c0d721dd474dde58ddc7394ff01d1b255aa758a5937aec74a
Sep 10 02:47:59.697681 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:59.697554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5" event={"ID":"9c53091c-e6fc-48c8-a734-bc27cffe7c7a","Type":"ContainerStarted","Data":"78260e51f638d87fb3133d359701b0696bbc1621496f3f285784f68a24f1957e"}
Sep 10 02:47:59.697681 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:59.697622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5" event={"ID":"9c53091c-e6fc-48c8-a734-bc27cffe7c7a","Type":"ContainerStarted","Data":"52319e60550af93853314da589a190a747669ec8db958efd17f9f7ed3650e168"}
Sep 10 02:47:59.698968 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:59.698941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerStarted","Data":"7f778e16dd86528c0d721dd474dde58ddc7394ff01d1b255aa758a5937aec74a"}
Sep 10 02:47:59.701080 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:59.701053    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e5bb6c7-94e4-4ea5-bb1f-34c41b535988" containerID="7d8f73ba69cc6595c846f1465f446660ca74f0e417095e59d040af59fc70800c" exitCode=0
Sep 10 02:47:59.701205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:47:59.701115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9svq4" event={"ID":"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988","Type":"ContainerDied","Data":"7d8f73ba69cc6595c846f1465f446660ca74f0e417095e59d040af59fc70800c"}
Sep 10 02:48:00.484621 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.484581    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"]
Sep 10 02:48:00.510338 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.510305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"]
Sep 10 02:48:00.510478 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.510407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.513540 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.513516    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 02:48:00.513718 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.513697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 02:48:00.513836 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.513812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cwktt\""
Sep 10 02:48:00.513941 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.513705    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 02:48:00.513941 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.513910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-85d890437aedh\""
Sep 10 02:48:00.514056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.514033    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 02:48:00.514097 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.514061    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 02:48:00.539386 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.539543 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59crs\" (UniqueName: \"kubernetes.io/projected/69861a10-4283-4df8-9d4e-70985d88b43b-kube-api-access-59crs\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.539543 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69861a10-4283-4df8-9d4e-70985d88b43b-metrics-client-ca\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.539644 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.539644 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.539753 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-grpc-tls\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.539753 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.539753 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.539717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-tls\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640355 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640355 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-grpc-tls\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-tls\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59crs\" (UniqueName: \"kubernetes.io/projected/69861a10-4283-4df8-9d4e-70985d88b43b-kube-api-access-59crs\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.640581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.640579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69861a10-4283-4df8-9d4e-70985d88b43b-metrics-client-ca\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.641463 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.641386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69861a10-4283-4df8-9d4e-70985d88b43b-metrics-client-ca\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.643652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.643618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.644104 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.644077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-tls\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.644826 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.644764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.645340 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.645310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-grpc-tls\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.645905 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.645882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.646822 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.646798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69861a10-4283-4df8-9d4e-70985d88b43b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.649948 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.649926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59crs\" (UniqueName: \"kubernetes.io/projected/69861a10-4283-4df8-9d4e-70985d88b43b-kube-api-access-59crs\") pod \"thanos-querier-5dd7d6df9c-d22nb\" (UID: \"69861a10-4283-4df8-9d4e-70985d88b43b\") " pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:00.821303 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:00.821222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:01.512906 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.512834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"]
Sep 10 02:48:01.515553 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:48:01.515524    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69861a10_4283_4df8_9d4e_70985d88b43b.slice/crio-cff6d7cc06d75ea7ba74408656e7a1f30750f1add74d7deadac4e8075944c760 WatchSource:0}: Error finding container cff6d7cc06d75ea7ba74408656e7a1f30750f1add74d7deadac4e8075944c760: Status 404 returned error can't find the container with id cff6d7cc06d75ea7ba74408656e7a1f30750f1add74d7deadac4e8075944c760
Sep 10 02:48:01.713768 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.713721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5" event={"ID":"9c53091c-e6fc-48c8-a734-bc27cffe7c7a","Type":"ContainerStarted","Data":"03a1c73c2c678a4d255e65e92dde08af14edd81be334baa9b3593be794771a9f"}
Sep 10 02:48:01.718485 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.718456    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerID="d5f87c4d7fa258fc2ee54aff5b34685f54775d380aae4406e4796233ef19b81d" exitCode=0
Sep 10 02:48:01.718623 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.718492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"d5f87c4d7fa258fc2ee54aff5b34685f54775d380aae4406e4796233ef19b81d"}
Sep 10 02:48:01.720190 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.720151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" event={"ID":"69861a10-4283-4df8-9d4e-70985d88b43b","Type":"ContainerStarted","Data":"cff6d7cc06d75ea7ba74408656e7a1f30750f1add74d7deadac4e8075944c760"}
Sep 10 02:48:01.722712 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.722693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9svq4" event={"ID":"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988","Type":"ContainerStarted","Data":"dae0f6f5e68f2f0e4a7f034731e2dd56e161153d293d1fd0d9804ae1b4241aa1"}
Sep 10 02:48:01.722832 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.722742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9svq4" event={"ID":"5e5bb6c7-94e4-4ea5-bb1f-34c41b535988","Type":"ContainerStarted","Data":"f0b3b06a5549a8a42e21153cef2336372d1364ac3ca4822a5e1bb809a9e21641"}
Sep 10 02:48:01.724932 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.724913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5" event={"ID":"52a23e89-45ac-4a22-a437-5da898f454d6","Type":"ContainerStarted","Data":"c5db36280ea010cbeba596b532080a5f879a624a1e40ad59c227bdf5546c94eb"}
Sep 10 02:48:01.725011 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.724956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5" event={"ID":"52a23e89-45ac-4a22-a437-5da898f454d6","Type":"ContainerStarted","Data":"f1aa22c84e8d60503da9011f6c0731af1fbbc15a3c81b88eef234a4dd8d8410f"}
Sep 10 02:48:01.725011 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.724993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5" event={"ID":"52a23e89-45ac-4a22-a437-5da898f454d6","Type":"ContainerStarted","Data":"cb4c1dcb30be6225d1ac85a0060b2c048b9fef68821e71d4ae5f7e2bb676de6a"}
Sep 10 02:48:01.732552 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.732478    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s77n5" podStartSLOduration=2.276126843 podStartE2EDuration="4.73246477s" totalImagesPullingTime="2.456337927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:01.730997379 +0000 UTC m=+78.880897080" watchObservedRunningTime="2026-09-10 02:48:01.73246477 +0000 UTC m=+78.882364467"
Sep 10 02:48:01.754888 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.754832    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9svq4" podStartSLOduration=3.822916762 podStartE2EDuration="4.754812178s" totalImagesPullingTime="855.119504ms" totalInitContainerRuntime="76.775912ms" isStatefulPod=true podCreationTimestamp="2026-09-10 02:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:01.752804175 +0000 UTC m=+78.902703874" watchObservedRunningTime="2026-09-10 02:48:01.754812178 +0000 UTC m=+78.904711877"
Sep 10 02:48:01.799056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:01.799009    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6fmh5" podStartSLOduration=2.233506908 podStartE2EDuration="4.798993769s" totalImagesPullingTime="2.565486861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:01.79855337 +0000 UTC m=+78.948453068" watchObservedRunningTime="2026-09-10 02:48:01.798993769 +0000 UTC m=+78.948893466"
Sep 10 02:48:03.883983 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.883960    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:48:03.887440 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.886856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.890038 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.890020    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r8n96\""
Sep 10 02:48:03.890204 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.890182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 02:48:03.890348 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.890118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 02:48:03.890460 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.890442    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 02:48:03.891987 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.891280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 02:48:03.891987 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.891591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 02:48:03.891987 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.891834    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 02:48:03.891987 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.891870    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 02:48:03.892301 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.892080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aci631d9j92ki\""
Sep 10 02:48:03.892679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.892482    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 02:48:03.892679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.892636    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 02:48:03.893210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.892664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 02:48:03.893210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.892765    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 02:48:03.893210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.892857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 02:48:03.893210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.892861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 02:48:03.895410 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.894889    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 02:48:03.898578 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.898558    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 02:48:03.914545 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.914513    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:48:03.972495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-web-config\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-config-out\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms494\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-kube-api-access-ms494\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.972812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.973107 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:03.973107 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:03.972892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-config\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073443 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073780 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-config\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073780 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-web-config\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073780 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073780 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.073780 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.073981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-config-out\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms494\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-kube-api-access-ms494\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074523 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.074625 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.075136 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.075136 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.074832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.077141 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.076832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.077265 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.077146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-web-config\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.077265 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.077146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-config\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.077625 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.077573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.077625 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.077610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.079449 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.079357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.079564 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.079450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.079631 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.079578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.079712 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.079686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.079804 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.079737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.080238 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.080216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.081505 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.081482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-config-out\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.081627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.081608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.085288 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.085268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms494\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-kube-api-access-ms494\") pod \"prometheus-k8s-0\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.198733 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.198675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:04.488071 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.487914    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:48:04.492794 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:48:04.492756    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062323b5_e9f4_4961_b44b_6a804ef902db.slice/crio-904e6fd727eae4692d331a3bbd7ca4ee512e06b6c0e0f9e7f48a3ca735245228 WatchSource:0}: Error finding container 904e6fd727eae4692d331a3bbd7ca4ee512e06b6c0e0f9e7f48a3ca735245228: Status 404 returned error can't find the container with id 904e6fd727eae4692d331a3bbd7ca4ee512e06b6c0e0f9e7f48a3ca735245228
Sep 10 02:48:04.747504 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.747425    2580 generic.go:356] "Generic (PLEG): container finished" podID="062323b5-e9f4-4961-b44b-6a804ef902db" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b" exitCode=0
Sep 10 02:48:04.747662 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.747509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"}
Sep 10 02:48:04.747662 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.747617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerStarted","Data":"904e6fd727eae4692d331a3bbd7ca4ee512e06b6c0e0f9e7f48a3ca735245228"}
Sep 10 02:48:04.750295 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.750263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" event={"ID":"69861a10-4283-4df8-9d4e-70985d88b43b","Type":"ContainerStarted","Data":"a2bcbbf0afb7be4673701bbc0fb09df7d9cbf5a617615f252010d0f94cc97e88"}
Sep 10 02:48:04.750431 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.750338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" event={"ID":"69861a10-4283-4df8-9d4e-70985d88b43b","Type":"ContainerStarted","Data":"edf97bc2b92a68aa1c1c3dc0b2f55e5c4a7e20406ae300860b85346029572ab8"}
Sep 10 02:48:04.750431 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.750380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" event={"ID":"69861a10-4283-4df8-9d4e-70985d88b43b","Type":"ContainerStarted","Data":"0f7191939a28f5deb1a03333df8aece5ab52db4c8df030b64eb0c86fe8f541a5"}
Sep 10 02:48:04.759645 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.759617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerStarted","Data":"eb207a7460c7860dcdcab4d744c80d43e0ab94ba163fad1000edb5dd1d042382"}
Sep 10 02:48:04.759746 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.759693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerStarted","Data":"de37a14d0d5dfcd6791563523c2e49fe902dcd9faab26b5d22ac4ffb61340727"}
Sep 10 02:48:04.759746 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.759725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerStarted","Data":"7616873d26871adb0946335d8bfdfc9bdaee22efc5331c149f1e7702f22ca6ca"}
Sep 10 02:48:04.759831 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.759748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerStarted","Data":"79487dcd16025e999216cae63503a68a2b8ed6345f6ed902d203dc89675a8aa7"}
Sep 10 02:48:04.759831 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:04.759772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerStarted","Data":"c5185606dc0b88893d32e78b582f831290a62dfbc2e59e954c3c1bfd38635e82"}
Sep 10 02:48:05.628715 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.628684    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-556d97688d-77wr5"
Sep 10 02:48:05.768061 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.767802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" event={"ID":"69861a10-4283-4df8-9d4e-70985d88b43b","Type":"ContainerStarted","Data":"55547071313e9ebb6a1e5997e675fa4a2ef5f13bdcefb91d32a102273702c715"}
Sep 10 02:48:05.768061 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.767884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" event={"ID":"69861a10-4283-4df8-9d4e-70985d88b43b","Type":"ContainerStarted","Data":"5d5e19b78f97bb3e5c050b5b889fe4611a37ee3b6831a0343a783226135586dc"}
Sep 10 02:48:05.768061 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.767925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:05.768061 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.767964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" event={"ID":"69861a10-4283-4df8-9d4e-70985d88b43b","Type":"ContainerStarted","Data":"07511816f7c6938bfb936942bb44d9ce8fd74c50228f9c94acd9603690b8f560"}
Sep 10 02:48:05.772202 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.772147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerStarted","Data":"a3c3ef90038ad1489b3a5b5c3ff3f481405613c399968a0a7d0812c706c8d1e3"}
Sep 10 02:48:05.792943 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.792897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb" podStartSLOduration=2.619387369 podStartE2EDuration="5.792878651s" totalImagesPullingTime="3.173491282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:05.792814823 +0000 UTC m=+82.942714523" watchObservedRunningTime="2026-09-10 02:48:05.792878651 +0000 UTC m=+82.942778327"
Sep 10 02:48:05.824718 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:05.824676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.913746412 podStartE2EDuration="7.824657871s" totalImagesPullingTime="4.882911596s" totalInitContainerRuntime="27.999863ms" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:05.823302541 +0000 UTC m=+82.973202242" watchObservedRunningTime="2026-09-10 02:48:05.824657871 +0000 UTC m=+82.974557595"
Sep 10 02:48:08.732468 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.732420    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb" podUID="289759e2-95bd-4ca8-910b-e6d169232a1c" containerName="registry" containerID="cri-o://f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d" gracePeriod=30
Sep 10 02:48:08.788568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.788540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerStarted","Data":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"}
Sep 10 02:48:08.788686 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.788603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerStarted","Data":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"}
Sep 10 02:48:08.788686 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.788630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerStarted","Data":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"}
Sep 10 02:48:08.788686 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.788660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerStarted","Data":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"}
Sep 10 02:48:08.788686 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.788684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerStarted","Data":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"}
Sep 10 02:48:08.788807 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.788711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerStarted","Data":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"}
Sep 10 02:48:08.823442 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.823386    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.76119449 podStartE2EDuration="5.823368444s" totalImagesPullingTime="3.037361401s" totalInitContainerRuntime="24.812553ms" isStatefulPod=false podCreationTimestamp="2026-09-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:08.822050586 +0000 UTC m=+85.971950285" watchObservedRunningTime="2026-09-10 02:48:08.823368444 +0000 UTC m=+85.973268144"
Sep 10 02:48:08.979209 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:08.979175    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:48:09.021919 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.021851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-bound-sa-token\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.021920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-installation-pull-secrets\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.021960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sznm\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-kube-api-access-5sznm\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.021989    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.022012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-image-registry-private-configuration\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.022037    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/289759e2-95bd-4ca8-910b-e6d169232a1c-ca-trust-extracted\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022337 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.022060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-certificates\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022337 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.022100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-trusted-ca\") pod \"289759e2-95bd-4ca8-910b-e6d169232a1c\" (UID: \"289759e2-95bd-4ca8-910b-e6d169232a1c\") "
Sep 10 02:48:09.022591 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.022560    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:48:09.022670 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.022640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:48:09.024658 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.024610    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-kube-api-access-5sznm" (OuterVolumeSpecName: "kube-api-access-5sznm") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "kube-api-access-5sznm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:48:09.024877 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.024848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:48:09.024977 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.024956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:48:09.025028 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.024987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:48:09.025115 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.025100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:48:09.033368 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.033343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289759e2-95bd-4ca8-910b-e6d169232a1c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "289759e2-95bd-4ca8-910b-e6d169232a1c" (UID: "289759e2-95bd-4ca8-910b-e6d169232a1c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:48:09.123585 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123558    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-image-registry-private-configuration\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.123585 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123581    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/289759e2-95bd-4ca8-910b-e6d169232a1c-ca-trust-extracted\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.123703 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123592    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-certificates\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.123703 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123602    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/289759e2-95bd-4ca8-910b-e6d169232a1c-trusted-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.123703 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123610    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-bound-sa-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.123703 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123618    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/289759e2-95bd-4ca8-910b-e6d169232a1c-installation-pull-secrets\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.123703 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123626    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sznm\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-kube-api-access-5sznm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.123703 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.123635    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/289759e2-95bd-4ca8-910b-e6d169232a1c-registry-tls\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:48:09.199412 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.199384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:48:09.794180 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.794128    2580 generic.go:356] "Generic (PLEG): container finished" podID="289759e2-95bd-4ca8-910b-e6d169232a1c" containerID="f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d" exitCode=0
Sep 10 02:48:09.794567 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.794206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"
Sep 10 02:48:09.794567 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.794205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb" event={"ID":"289759e2-95bd-4ca8-910b-e6d169232a1c","Type":"ContainerDied","Data":"f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d"}
Sep 10 02:48:09.794567 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.794260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb" event={"ID":"289759e2-95bd-4ca8-910b-e6d169232a1c","Type":"ContainerDied","Data":"8e7a4a14860ae6b740bf1a55db43a979c1a0257dc207ba34bca3010e541e867d"}
Sep 10 02:48:09.794567 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.794280    2580 scope.go:117] "RemoveContainer" containerID="f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d"
Sep 10 02:48:09.803308 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.803292    2580 scope.go:117] "RemoveContainer" containerID="f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d"
Sep 10 02:48:09.803564 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:48:09.803543    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d\": container with ID starting with f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d not found: ID does not exist" containerID="f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d"
Sep 10 02:48:09.803617 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.803571    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d"} err="failed to get container status \"f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d\": rpc error: code = NotFound desc = could not find container \"f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d\": container with ID starting with f05dbef231ec15fbbb5a5787fb03144d3d276427554c9d79b02cf368ff795a3d not found: ID does not exist"
Sep 10 02:48:09.814094 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.814066    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"]
Sep 10 02:48:09.817649 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:09.817629    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dd7f7dbdd-9xjwb"]
Sep 10 02:48:11.323737 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:11.323704    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289759e2-95bd-4ca8-910b-e6d169232a1c" path="/var/lib/kubelet/pods/289759e2-95bd-4ca8-910b-e6d169232a1c/volumes"
Sep 10 02:48:11.782941 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:11.782916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dd7d6df9c-d22nb"
Sep 10 02:48:21.660575 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:48:21.660540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z7xf8"
Sep 10 02:49:04.199413 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:04.199382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:04.217849 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:04.217819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:05.048581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:05.048556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:17.929319 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:17.929278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:49:17.929797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:17.929654    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="alertmanager" containerID="cri-o://c5185606dc0b88893d32e78b582f831290a62dfbc2e59e954c3c1bfd38635e82" gracePeriod=120
Sep 10 02:49:17.929797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:17.929687    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-metric" containerID="cri-o://eb207a7460c7860dcdcab4d744c80d43e0ab94ba163fad1000edb5dd1d042382" gracePeriod=120
Sep 10 02:49:17.929797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:17.929716    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="prom-label-proxy" containerID="cri-o://a3c3ef90038ad1489b3a5b5c3ff3f481405613c399968a0a7d0812c706c8d1e3" gracePeriod=120
Sep 10 02:49:17.929797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:17.929730    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-web" containerID="cri-o://7616873d26871adb0946335d8bfdfc9bdaee22efc5331c149f1e7702f22ca6ca" gracePeriod=120
Sep 10 02:49:17.929797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:17.929777    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="config-reloader" containerID="cri-o://79487dcd16025e999216cae63503a68a2b8ed6345f6ed902d203dc89675a8aa7" gracePeriod=120
Sep 10 02:49:17.930025 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:17.929737    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy" containerID="cri-o://de37a14d0d5dfcd6791563523c2e49fe902dcd9faab26b5d22ac4ffb61340727" gracePeriod=120
Sep 10 02:49:18.085367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085330    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerID="a3c3ef90038ad1489b3a5b5c3ff3f481405613c399968a0a7d0812c706c8d1e3" exitCode=0
Sep 10 02:49:18.085367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085357    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerID="de37a14d0d5dfcd6791563523c2e49fe902dcd9faab26b5d22ac4ffb61340727" exitCode=0
Sep 10 02:49:18.085367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085366    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerID="79487dcd16025e999216cae63503a68a2b8ed6345f6ed902d203dc89675a8aa7" exitCode=0
Sep 10 02:49:18.085545 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085374    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerID="c5185606dc0b88893d32e78b582f831290a62dfbc2e59e954c3c1bfd38635e82" exitCode=0
Sep 10 02:49:18.085545 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"a3c3ef90038ad1489b3a5b5c3ff3f481405613c399968a0a7d0812c706c8d1e3"}
Sep 10 02:49:18.085545 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"de37a14d0d5dfcd6791563523c2e49fe902dcd9faab26b5d22ac4ffb61340727"}
Sep 10 02:49:18.085545 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"79487dcd16025e999216cae63503a68a2b8ed6345f6ed902d203dc89675a8aa7"}
Sep 10 02:49:18.085545 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:18.085506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"c5185606dc0b88893d32e78b582f831290a62dfbc2e59e954c3c1bfd38635e82"}
Sep 10 02:49:19.094072 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.094011    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerID="eb207a7460c7860dcdcab4d744c80d43e0ab94ba163fad1000edb5dd1d042382" exitCode=0
Sep 10 02:49:19.094072 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.094043    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerID="7616873d26871adb0946335d8bfdfc9bdaee22efc5331c149f1e7702f22ca6ca" exitCode=0
Sep 10 02:49:19.094526 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.094184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"eb207a7460c7860dcdcab4d744c80d43e0ab94ba163fad1000edb5dd1d042382"}
Sep 10 02:49:19.094526 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.094266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"7616873d26871adb0946335d8bfdfc9bdaee22efc5331c149f1e7702f22ca6ca"}
Sep 10 02:49:19.173809 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.173786    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:19.274504 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274437    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-volume\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274504 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzbd7\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-kube-api-access-wzbd7\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-web\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-main-tls\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-metrics-client-ca\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-web-config\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-tls-assets\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274971 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-main-db\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274971 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-out\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274971 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-cluster-tls-config\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274971 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-trusted-ca-bundle\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274971 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy\") pod \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\" (UID: \"bdb3a3be-3403-401a-a89c-a71103ba9cc4\") "
Sep 10 02:49:19.274971 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.274942    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:19.275283 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.275125    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-metrics-client-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.275283 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.275193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:49:19.275821 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.275610    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:19.277445 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.277417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:19.277538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.277476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:49:19.277538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.277508    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-kube-api-access-wzbd7" (OuterVolumeSpecName: "kube-api-access-wzbd7") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "kube-api-access-wzbd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:49:19.277641 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.277596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-out" (OuterVolumeSpecName: "config-out") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:49:19.277746 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.277716    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:19.277950 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.277927    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-volume" (OuterVolumeSpecName: "config-volume") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:19.278082 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.278064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:19.278159 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.278144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:19.295675 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.295620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:19.311327 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.311300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-web-config" (OuterVolumeSpecName: "web-config") pod "bdb3a3be-3403-401a-a89c-a71103ba9cc4" (UID: "bdb3a3be-3403-401a-a89c-a71103ba9cc4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:19.375720 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375699    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-out\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375720 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375719    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-cluster-tls-config\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375731    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375741    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375749    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375758    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbd7\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-kube-api-access-wzbd7\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375768    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375777    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-main-tls\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375785    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-web-config\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375794    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdb3a3be-3403-401a-a89c-a71103ba9cc4-tls-assets\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375802    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bdb3a3be-3403-401a-a89c-a71103ba9cc4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:19.375850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:19.375810    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bdb3a3be-3403-401a-a89c-a71103ba9cc4-alertmanager-main-db\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:20.101280 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.101245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bdb3a3be-3403-401a-a89c-a71103ba9cc4","Type":"ContainerDied","Data":"7f778e16dd86528c0d721dd474dde58ddc7394ff01d1b255aa758a5937aec74a"}
Sep 10 02:49:20.101624 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.101253    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.101624 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.101337    2580 scope.go:117] "RemoveContainer" containerID="a3c3ef90038ad1489b3a5b5c3ff3f481405613c399968a0a7d0812c706c8d1e3"
Sep 10 02:49:20.108631 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.108614    2580 scope.go:117] "RemoveContainer" containerID="eb207a7460c7860dcdcab4d744c80d43e0ab94ba163fad1000edb5dd1d042382"
Sep 10 02:49:20.117389 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.117301    2580 scope.go:117] "RemoveContainer" containerID="de37a14d0d5dfcd6791563523c2e49fe902dcd9faab26b5d22ac4ffb61340727"
Sep 10 02:49:20.123936 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.123919    2580 scope.go:117] "RemoveContainer" containerID="7616873d26871adb0946335d8bfdfc9bdaee22efc5331c149f1e7702f22ca6ca"
Sep 10 02:49:20.131394 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.131281    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:49:20.131453 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.131417    2580 scope.go:117] "RemoveContainer" containerID="79487dcd16025e999216cae63503a68a2b8ed6345f6ed902d203dc89675a8aa7"
Sep 10 02:49:20.139176 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.139146    2580 scope.go:117] "RemoveContainer" containerID="c5185606dc0b88893d32e78b582f831290a62dfbc2e59e954c3c1bfd38635e82"
Sep 10 02:49:20.139776 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.139753    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:49:20.145832 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.145816    2580 scope.go:117] "RemoveContainer" containerID="d5f87c4d7fa258fc2ee54aff5b34685f54775d380aae4406e4796233ef19b81d"
Sep 10 02:49:20.172204 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172182    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:49:20.172490 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172477    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-metric"
Sep 10 02:49:20.172530 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172492    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-metric"
Sep 10 02:49:20.172530 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172509    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="init-config-reloader"
Sep 10 02:49:20.172530 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172518    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="init-config-reloader"
Sep 10 02:49:20.172530 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172528    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="alertmanager"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172534    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="alertmanager"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172542    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-web"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-web"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172556    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="prom-label-proxy"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172561    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="prom-label-proxy"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172568    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172573    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172580    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="config-reloader"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172585    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="config-reloader"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172593    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289759e2-95bd-4ca8-910b-e6d169232a1c" containerName="registry"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172601    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="289759e2-95bd-4ca8-910b-e6d169232a1c" containerName="registry"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172648    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="alertmanager"
Sep 10 02:49:20.172652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172655    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy"
Sep 10 02:49:20.172999 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172663    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-web"
Sep 10 02:49:20.172999 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172669    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="kube-rbac-proxy-metric"
Sep 10 02:49:20.172999 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172679    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="config-reloader"
Sep 10 02:49:20.172999 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172687    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="289759e2-95bd-4ca8-910b-e6d169232a1c" containerName="registry"
Sep 10 02:49:20.172999 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.172692    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" containerName="prom-label-proxy"
Sep 10 02:49:20.178347 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.177690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.181438 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.181414    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 02:49:20.181735 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.181715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 02:49:20.181803 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.181781    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 02:49:20.182030 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.182014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 02:49:20.182106 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.182026    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dlmdm\""
Sep 10 02:49:20.182158 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.182014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 02:49:20.182243 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.182228    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 02:49:20.182357 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.182337    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 02:49:20.182462 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.182406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 02:49:20.186674 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.186656    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 02:49:20.194870 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.194849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:49:20.282331 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282331 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e99e2739-8452-4ba4-a4e3-3476cb3aa746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282472 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282472 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e99e2739-8452-4ba4-a4e3-3476cb3aa746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282472 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282472 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99e2739-8452-4ba4-a4e3-3476cb3aa746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-web-config\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e99e2739-8452-4ba4-a4e3-3476cb3aa746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e99e2739-8452-4ba4-a4e3-3476cb3aa746-config-out\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282785 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-config-volume\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.282785 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.282645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fns8c\" (UniqueName: \"kubernetes.io/projected/e99e2739-8452-4ba4-a4e3-3476cb3aa746-kube-api-access-fns8c\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383544 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e99e2739-8452-4ba4-a4e3-3476cb3aa746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e99e2739-8452-4ba4-a4e3-3476cb3aa746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99e2739-8452-4ba4-a4e3-3476cb3aa746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-web-config\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.383847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.383777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e99e2739-8452-4ba4-a4e3-3476cb3aa746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.384095 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.384008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e99e2739-8452-4ba4-a4e3-3476cb3aa746-config-out\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.384095 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.384087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-config-volume\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.384220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.384122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fns8c\" (UniqueName: \"kubernetes.io/projected/e99e2739-8452-4ba4-a4e3-3476cb3aa746-kube-api-access-fns8c\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.384668 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.384620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e99e2739-8452-4ba4-a4e3-3476cb3aa746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.386921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.384935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99e2739-8452-4ba4-a4e3-3476cb3aa746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.386921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.386631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.386921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.386655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.386921 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.386878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.387192 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.387140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-web-config\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.387451 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.387424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e99e2739-8452-4ba4-a4e3-3476cb3aa746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.387451 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.387433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e99e2739-8452-4ba4-a4e3-3476cb3aa746-config-out\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.387698 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.387672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.388121 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.387885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-config-volume\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.388410 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.388392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e99e2739-8452-4ba4-a4e3-3476cb3aa746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.389837 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.389817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e99e2739-8452-4ba4-a4e3-3476cb3aa746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.393815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.393792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fns8c\" (UniqueName: \"kubernetes.io/projected/e99e2739-8452-4ba4-a4e3-3476cb3aa746-kube-api-access-fns8c\") pod \"alertmanager-main-0\" (UID: \"e99e2739-8452-4ba4-a4e3-3476cb3aa746\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.487800 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.487779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 02:49:20.771451 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:20.771427    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 02:49:20.773402 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:49:20.773377    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99e2739_8452_4ba4_a4e3_3476cb3aa746.slice/crio-5cf2430d97ecc4aaaeb8dcbea99767d5eea342f52cc1dc24df3dfd13f6996a1f WatchSource:0}: Error finding container 5cf2430d97ecc4aaaeb8dcbea99767d5eea342f52cc1dc24df3dfd13f6996a1f: Status 404 returned error can't find the container with id 5cf2430d97ecc4aaaeb8dcbea99767d5eea342f52cc1dc24df3dfd13f6996a1f
Sep 10 02:49:21.107060 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:21.107029    2580 generic.go:356] "Generic (PLEG): container finished" podID="e99e2739-8452-4ba4-a4e3-3476cb3aa746" containerID="c10587c77ec0044c8be823b223ef0e0f218542dd9bd84f1f24139aa6e2dd85bd" exitCode=0
Sep 10 02:49:21.107479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:21.107117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerDied","Data":"c10587c77ec0044c8be823b223ef0e0f218542dd9bd84f1f24139aa6e2dd85bd"}
Sep 10 02:49:21.107479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:21.107221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerStarted","Data":"5cf2430d97ecc4aaaeb8dcbea99767d5eea342f52cc1dc24df3dfd13f6996a1f"}
Sep 10 02:49:21.326059 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:21.326033    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb3a3be-3403-401a-a89c-a71103ba9cc4" path="/var/lib/kubelet/pods/bdb3a3be-3403-401a-a89c-a71103ba9cc4/volumes"
Sep 10 02:49:22.115274 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.115239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerStarted","Data":"a14a34930af8991645a2ee41748a6204749c56988790a742e0b39b72c6a1839f"}
Sep 10 02:49:22.115699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.115303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerStarted","Data":"cd402f8298b6ce3521eb40031cee4a970782e76ce19d9c84606ed3764d41bf1a"}
Sep 10 02:49:22.115699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.115330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerStarted","Data":"a47882b7c3c960088eee30eaf4568f7b39eef184e26f851a1d9e4441e3317dd1"}
Sep 10 02:49:22.115699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.115356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerStarted","Data":"43a84fc9d58a66ad4d91c2b51040310169e945e5492c2f44510601f1603ed8a8"}
Sep 10 02:49:22.115699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.115379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerStarted","Data":"99bbab2869e3ad985f5226eab7c09b29979ca7b420ff6d9fe523fc7574c97f7d"}
Sep 10 02:49:22.115699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.115401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e99e2739-8452-4ba4-a4e3-3476cb3aa746","Type":"ContainerStarted","Data":"ee4f9bb0d5be02b9e9ae97e056aad0f992b171cd9034325f7978a902bf9ad0ae"}
Sep 10 02:49:22.155590 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.155540    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.138975363 podStartE2EDuration="2.155524264s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.548901ms" isStatefulPod=false podCreationTimestamp="2026-09-10 02:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:49:22.152968165 +0000 UTC m=+159.302867863" watchObservedRunningTime="2026-09-10 02:49:22.155524264 +0000 UTC m=+159.305423961"
Sep 10 02:49:22.184575 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.183562    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:49:22.184575 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.184085    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="prometheus" containerID="cri-o://5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343" gracePeriod=600
Sep 10 02:49:22.184575 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.184302    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-thanos" containerID="cri-o://24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc" gracePeriod=600
Sep 10 02:49:22.184575 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.184319    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="thanos-sidecar" containerID="cri-o://e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7" gracePeriod=600
Sep 10 02:49:22.184575 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.184312    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy" containerID="cri-o://f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00" gracePeriod=600
Sep 10 02:49:22.184915 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.184627    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="config-reloader" containerID="cri-o://01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d" gracePeriod=600
Sep 10 02:49:22.184915 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.184735    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-web" containerID="cri-o://963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f" gracePeriod=600
Sep 10 02:49:22.430441 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.430418    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:22.502382 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-db\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502382 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-metrics-client-ca\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502598 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-1\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502598 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502598 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-thanos-prometheus-http-client-file\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502598 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-0\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502598 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502552    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-tls\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502598 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-kube-rbac-proxy\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502674    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-2\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-serving-certs-ca-bundle\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502727    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-config\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-tls-assets\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms494\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-kube-api-access-ms494\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-kubelet-serving-ca-bundle\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:22.502885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-config-out\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.503374 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502912    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-metrics-client-certs\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.503374 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-grpc-tls\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.503374 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-web-config\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.503374 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.502995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-trusted-ca-bundle\") pod \"062323b5-e9f4-4961-b44b-6a804ef902db\" (UID: \"062323b5-e9f4-4961-b44b-6a804ef902db\") "
Sep 10 02:49:22.503374 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.503087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:22.503615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.503288    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-metrics-client-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.503615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.503400    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.503615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.503510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:22.503764 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.503719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:22.504153 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.504124    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:49:22.505273 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.505243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:22.505273 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.505250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:22.506077 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.506046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:49:22.506188 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.506131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.506635 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.506609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.506793 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.506770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:49:22.507009 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.506974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-config" (OuterVolumeSpecName: "config") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.507101 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.507067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.507208 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.507187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.507620 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.507594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.507895 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.507867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-kube-api-access-ms494" (OuterVolumeSpecName: "kube-api-access-ms494") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "kube-api-access-ms494". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:49:22.507963 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.507910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.508055 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.508023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-config-out" (OuterVolumeSpecName: "config-out") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:49:22.508133 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.508090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.520127 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.520100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-web-config" (OuterVolumeSpecName: "web-config") pod "062323b5-e9f4-4961-b44b-6a804ef902db" (UID: "062323b5-e9f4-4961-b44b-6a804ef902db"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:49:22.604516 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604492    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-tls-assets\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604516 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604513    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms494\" (UniqueName: \"kubernetes.io/projected/062323b5-e9f4-4961-b44b-6a804ef902db-kube-api-access-ms494\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604525    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604534    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-config-out\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604542    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-metrics-client-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604550    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-grpc-tls\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604557    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-web-config\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604565    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604574    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-db\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604582    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604592    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604600    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604608    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604617    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-kube-rbac-proxy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604625    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604634    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604642    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/062323b5-e9f4-4961-b44b-6a804ef902db-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:22.604664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:22.604651    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/062323b5-e9f4-4961-b44b-6a804ef902db-config\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:49:23.122767 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122733    2580 generic.go:356] "Generic (PLEG): container finished" podID="062323b5-e9f4-4961-b44b-6a804ef902db" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc" exitCode=0
Sep 10 02:49:23.122767 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122757    2580 generic.go:356] "Generic (PLEG): container finished" podID="062323b5-e9f4-4961-b44b-6a804ef902db" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00" exitCode=0
Sep 10 02:49:23.122767 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122763    2580 generic.go:356] "Generic (PLEG): container finished" podID="062323b5-e9f4-4961-b44b-6a804ef902db" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f" exitCode=0
Sep 10 02:49:23.122767 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122770    2580 generic.go:356] "Generic (PLEG): container finished" podID="062323b5-e9f4-4961-b44b-6a804ef902db" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7" exitCode=0
Sep 10 02:49:23.122767 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122776    2580 generic.go:356] "Generic (PLEG): container finished" podID="062323b5-e9f4-4961-b44b-6a804ef902db" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d" exitCode=0
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122786    2580 generic.go:356] "Generic (PLEG): container finished" podID="062323b5-e9f4-4961-b44b-6a804ef902db" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343" exitCode=0
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"}
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"}
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122823    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"}
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"}
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122975    2580 scope.go:117] "RemoveContainer" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.122978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"}
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.123094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"}
Sep 10 02:49:23.123427 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.123127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"062323b5-e9f4-4961-b44b-6a804ef902db","Type":"ContainerDied","Data":"904e6fd727eae4692d331a3bbd7ca4ee512e06b6c0e0f9e7f48a3ca735245228"}
Sep 10 02:49:23.131039 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.131014    2580 scope.go:117] "RemoveContainer" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.138029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.138013    2580 scope.go:117] "RemoveContainer" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.144705 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.144690    2580 scope.go:117] "RemoveContainer" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.151442 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.151406    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:49:23.152293 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.152265    2580 scope.go:117] "RemoveContainer" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.153824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.153808    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:49:23.158746 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.158727    2580 scope.go:117] "RemoveContainer" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.165744 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.165728    2580 scope.go:117] "RemoveContainer" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.172323 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.172305    2580 scope.go:117] "RemoveContainer" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.172571 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:49:23.172550    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": container with ID starting with 24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc not found: ID does not exist" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.172638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.172584    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"} err="failed to get container status \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": rpc error: code = NotFound desc = could not find container \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": container with ID starting with 24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc not found: ID does not exist"
Sep 10 02:49:23.172638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.172607    2580 scope.go:117] "RemoveContainer" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.172843 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:49:23.172827    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": container with ID starting with f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00 not found: ID does not exist" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.172879 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.172851    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"} err="failed to get container status \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": rpc error: code = NotFound desc = could not find container \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": container with ID starting with f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00 not found: ID does not exist"
Sep 10 02:49:23.172879 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.172865    2580 scope.go:117] "RemoveContainer" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.173075 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:49:23.173056    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": container with ID starting with 963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f not found: ID does not exist" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.173119 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173081    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"} err="failed to get container status \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": rpc error: code = NotFound desc = could not find container \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": container with ID starting with 963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f not found: ID does not exist"
Sep 10 02:49:23.173119 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173097    2580 scope.go:117] "RemoveContainer" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.173346 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:49:23.173330    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": container with ID starting with e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7 not found: ID does not exist" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.173412 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173353    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"} err="failed to get container status \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": rpc error: code = NotFound desc = could not find container \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": container with ID starting with e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7 not found: ID does not exist"
Sep 10 02:49:23.173412 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173372    2580 scope.go:117] "RemoveContainer" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.173594 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:49:23.173579    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": container with ID starting with 01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d not found: ID does not exist" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.173632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173597    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"} err="failed to get container status \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": rpc error: code = NotFound desc = could not find container \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": container with ID starting with 01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d not found: ID does not exist"
Sep 10 02:49:23.173632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173612    2580 scope.go:117] "RemoveContainer" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.173800 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:49:23.173783    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": container with ID starting with 5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343 not found: ID does not exist" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.173860 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173806    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"} err="failed to get container status \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": rpc error: code = NotFound desc = could not find container \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": container with ID starting with 5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343 not found: ID does not exist"
Sep 10 02:49:23.173860 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.173824    2580 scope.go:117] "RemoveContainer" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.174033 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:49:23.174020    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": container with ID starting with 822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b not found: ID does not exist" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.174070 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174036    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"} err="failed to get container status \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": rpc error: code = NotFound desc = could not find container \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": container with ID starting with 822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b not found: ID does not exist"
Sep 10 02:49:23.174070 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174048    2580 scope.go:117] "RemoveContainer" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.174291 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174273    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"} err="failed to get container status \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": rpc error: code = NotFound desc = could not find container \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": container with ID starting with 24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc not found: ID does not exist"
Sep 10 02:49:23.174354 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174292    2580 scope.go:117] "RemoveContainer" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.174536 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"} err="failed to get container status \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": rpc error: code = NotFound desc = could not find container \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": container with ID starting with f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00 not found: ID does not exist"
Sep 10 02:49:23.174536 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174534    2580 scope.go:117] "RemoveContainer" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.174754 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174739    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"} err="failed to get container status \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": rpc error: code = NotFound desc = could not find container \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": container with ID starting with 963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f not found: ID does not exist"
Sep 10 02:49:23.174806 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174754    2580 scope.go:117] "RemoveContainer" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.174951 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174934    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"} err="failed to get container status \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": rpc error: code = NotFound desc = could not find container \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": container with ID starting with e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7 not found: ID does not exist"
Sep 10 02:49:23.174991 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.174951    2580 scope.go:117] "RemoveContainer" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.175147 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175132    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"} err="failed to get container status \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": rpc error: code = NotFound desc = could not find container \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": container with ID starting with 01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d not found: ID does not exist"
Sep 10 02:49:23.175206 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175146    2580 scope.go:117] "RemoveContainer" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.175390 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175376    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"} err="failed to get container status \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": rpc error: code = NotFound desc = could not find container \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": container with ID starting with 5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343 not found: ID does not exist"
Sep 10 02:49:23.175434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175390    2580 scope.go:117] "RemoveContainer" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.175580 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175562    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"} err="failed to get container status \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": rpc error: code = NotFound desc = could not find container \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": container with ID starting with 822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b not found: ID does not exist"
Sep 10 02:49:23.175626 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175580    2580 scope.go:117] "RemoveContainer" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.175800 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175781    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"} err="failed to get container status \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": rpc error: code = NotFound desc = could not find container \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": container with ID starting with 24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc not found: ID does not exist"
Sep 10 02:49:23.175867 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.175802    2580 scope.go:117] "RemoveContainer" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.176038 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176021    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"} err="failed to get container status \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": rpc error: code = NotFound desc = could not find container \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": container with ID starting with f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00 not found: ID does not exist"
Sep 10 02:49:23.176078 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176039    2580 scope.go:117] "RemoveContainer" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.176256 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176241    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"} err="failed to get container status \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": rpc error: code = NotFound desc = could not find container \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": container with ID starting with 963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f not found: ID does not exist"
Sep 10 02:49:23.176256 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176255    2580 scope.go:117] "RemoveContainer" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.176454 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176436    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"} err="failed to get container status \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": rpc error: code = NotFound desc = could not find container \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": container with ID starting with e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7 not found: ID does not exist"
Sep 10 02:49:23.176517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176455    2580 scope.go:117] "RemoveContainer" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.176679 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176662    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"} err="failed to get container status \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": rpc error: code = NotFound desc = could not find container \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": container with ID starting with 01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d not found: ID does not exist"
Sep 10 02:49:23.176720 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176680    2580 scope.go:117] "RemoveContainer" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.176880 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176863    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"} err="failed to get container status \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": rpc error: code = NotFound desc = could not find container \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": container with ID starting with 5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343 not found: ID does not exist"
Sep 10 02:49:23.176880 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.176879    2580 scope.go:117] "RemoveContainer" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.177085 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177061    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"} err="failed to get container status \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": rpc error: code = NotFound desc = could not find container \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": container with ID starting with 822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b not found: ID does not exist"
Sep 10 02:49:23.177127 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177085    2580 scope.go:117] "RemoveContainer" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.177366 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177336    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"} err="failed to get container status \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": rpc error: code = NotFound desc = could not find container \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": container with ID starting with 24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc not found: ID does not exist"
Sep 10 02:49:23.177366 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177363    2580 scope.go:117] "RemoveContainer" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.177599 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177582    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"} err="failed to get container status \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": rpc error: code = NotFound desc = could not find container \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": container with ID starting with f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00 not found: ID does not exist"
Sep 10 02:49:23.177599 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177599    2580 scope.go:117] "RemoveContainer" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.177847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177827    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"} err="failed to get container status \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": rpc error: code = NotFound desc = could not find container \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": container with ID starting with 963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f not found: ID does not exist"
Sep 10 02:49:23.177847 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.177846    2580 scope.go:117] "RemoveContainer" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.178159 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178139    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"} err="failed to get container status \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": rpc error: code = NotFound desc = could not find container \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": container with ID starting with e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7 not found: ID does not exist"
Sep 10 02:49:23.178261 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178159    2580 scope.go:117] "RemoveContainer" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.178477 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178453    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"} err="failed to get container status \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": rpc error: code = NotFound desc = could not find container \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": container with ID starting with 01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d not found: ID does not exist"
Sep 10 02:49:23.178538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178478    2580 scope.go:117] "RemoveContainer" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.178736 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178717    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"} err="failed to get container status \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": rpc error: code = NotFound desc = could not find container \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": container with ID starting with 5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343 not found: ID does not exist"
Sep 10 02:49:23.178736 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178735    2580 scope.go:117] "RemoveContainer" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.178953 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178936    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"} err="failed to get container status \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": rpc error: code = NotFound desc = could not find container \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": container with ID starting with 822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b not found: ID does not exist"
Sep 10 02:49:23.179001 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.178954    2580 scope.go:117] "RemoveContainer" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.179180 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179147    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"} err="failed to get container status \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": rpc error: code = NotFound desc = could not find container \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": container with ID starting with 24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc not found: ID does not exist"
Sep 10 02:49:23.179239 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179184    2580 scope.go:117] "RemoveContainer" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.179377 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179362    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"} err="failed to get container status \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": rpc error: code = NotFound desc = could not find container \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": container with ID starting with f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00 not found: ID does not exist"
Sep 10 02:49:23.179377 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179375    2580 scope.go:117] "RemoveContainer" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.179564 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179551    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"} err="failed to get container status \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": rpc error: code = NotFound desc = could not find container \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": container with ID starting with 963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f not found: ID does not exist"
Sep 10 02:49:23.179616 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179564    2580 scope.go:117] "RemoveContainer" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.179800 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179781    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"} err="failed to get container status \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": rpc error: code = NotFound desc = could not find container \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": container with ID starting with e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7 not found: ID does not exist"
Sep 10 02:49:23.179845 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179800    2580 scope.go:117] "RemoveContainer" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.180009 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.179994    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"} err="failed to get container status \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": rpc error: code = NotFound desc = could not find container \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": container with ID starting with 01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d not found: ID does not exist"
Sep 10 02:49:23.180046 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180009    2580 scope.go:117] "RemoveContainer" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.180216 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180201    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"} err="failed to get container status \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": rpc error: code = NotFound desc = could not find container \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": container with ID starting with 5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343 not found: ID does not exist"
Sep 10 02:49:23.180261 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180215    2580 scope.go:117] "RemoveContainer" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.180402 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180385    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"} err="failed to get container status \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": rpc error: code = NotFound desc = could not find container \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": container with ID starting with 822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b not found: ID does not exist"
Sep 10 02:49:23.180440 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180403    2580 scope.go:117] "RemoveContainer" containerID="24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"
Sep 10 02:49:23.180588 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180573    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc"} err="failed to get container status \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": rpc error: code = NotFound desc = could not find container \"24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc\": container with ID starting with 24bb131bee8ba7a9434356e3d924b2615999c8478bec0bff3acfc2fa06f124bc not found: ID does not exist"
Sep 10 02:49:23.180624 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180589    2580 scope.go:117] "RemoveContainer" containerID="f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"
Sep 10 02:49:23.180774 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180759    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00"} err="failed to get container status \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": rpc error: code = NotFound desc = could not find container \"f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00\": container with ID starting with f78e492c2118369d4c990ebe9a9fb7c578ca229e04bec20a481106503ad21d00 not found: ID does not exist"
Sep 10 02:49:23.180814 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180774    2580 scope.go:117] "RemoveContainer" containerID="963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"
Sep 10 02:49:23.180952 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180937    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f"} err="failed to get container status \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": rpc error: code = NotFound desc = could not find container \"963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f\": container with ID starting with 963d27e365e8f4b6a5a204417f0304c09743b0d5ae985a6aacbc2f2b8efa8a9f not found: ID does not exist"
Sep 10 02:49:23.180952 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.180951    2580 scope.go:117] "RemoveContainer" containerID="e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"
Sep 10 02:49:23.181146 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.181130    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7"} err="failed to get container status \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": rpc error: code = NotFound desc = could not find container \"e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7\": container with ID starting with e694a664c588fb381f8250846d71a04d53bed4e82f11988d1d4242c4045730e7 not found: ID does not exist"
Sep 10 02:49:23.181214 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.181147    2580 scope.go:117] "RemoveContainer" containerID="01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"
Sep 10 02:49:23.181361 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.181346    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d"} err="failed to get container status \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": rpc error: code = NotFound desc = could not find container \"01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d\": container with ID starting with 01d5800f7e133ffcc20a1fde121ba947cc27cbb6eddf959a304ef18f58beb85d not found: ID does not exist"
Sep 10 02:49:23.181397 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.181360    2580 scope.go:117] "RemoveContainer" containerID="5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"
Sep 10 02:49:23.181561 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.181543    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343"} err="failed to get container status \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": rpc error: code = NotFound desc = could not find container \"5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343\": container with ID starting with 5b303dc56de76734064a94be069b51510a94f874eeb58c3ab0f6fb7720830343 not found: ID does not exist"
Sep 10 02:49:23.181600 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.181562    2580 scope.go:117] "RemoveContainer" containerID="822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"
Sep 10 02:49:23.181782 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.181765    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b"} err="failed to get container status \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": rpc error: code = NotFound desc = could not find container \"822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b\": container with ID starting with 822db0262e2fed1d427b1f3a5df98f93601d2121f5dc74587a513fea310fda0b not found: ID does not exist"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.193807    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194244    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-thanos"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-thanos"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194274    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="thanos-sidecar"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194282    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="thanos-sidecar"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194291    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194299    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194310    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="init-config-reloader"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194318    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="init-config-reloader"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194327    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="config-reloader"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194335    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="config-reloader"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194347    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="prometheus"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194356    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="prometheus"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194377    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-web"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194385    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-web"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194455    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-web"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194466    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy-thanos"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194476    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="prometheus"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194486    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="config-reloader"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194494    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="thanos-sidecar"
Sep 10 02:49:23.196324 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.194505    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" containerName="kube-rbac-proxy"
Sep 10 02:49:23.199990 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.199975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.203786 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.203770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 02:49:23.203964 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.203950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 02:49:23.204098 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 02:49:23.204205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aci631d9j92ki\""
Sep 10 02:49:23.204272 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204244    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 02:49:23.204391 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204377    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 02:49:23.204561 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 02:49:23.204707 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204690    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 02:49:23.204988 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204973    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 02:49:23.205048 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.204972    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r8n96\""
Sep 10 02:49:23.205048 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.205012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 02:49:23.205048 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.205020    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 02:49:23.205625 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.205608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 02:49:23.205712 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.205640    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 02:49:23.205856 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.205843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 02:49:23.210485 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.210467    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 02:49:23.213236 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.213220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 02:49:23.218779 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.218760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:49:23.309680 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309680 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309680 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-config\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.309861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5m6\" (UniqueName: \"kubernetes.io/projected/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-kube-api-access-7p5m6\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.309999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.310038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.310067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.310091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.310237 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.310126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.323584 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.323562    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="062323b5-e9f4-4961-b44b-6a804ef902db" path="/var/lib/kubelet/pods/062323b5-e9f4-4961-b44b-6a804ef902db/volumes"
Sep 10 02:49:23.411379 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411485 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-config\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411485 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411485 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411485 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411747 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411747 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411747 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411747 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411747 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.411747 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5m6\" (UniqueName: \"kubernetes.io/projected/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-kube-api-access-7p5m6\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412029 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.411946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412384 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.412281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412769 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.412743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412899 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.412794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412899 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.412799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.412899 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.412860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.413056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.412906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.415197 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.414471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-config\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.415197 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.414512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.415197 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.414871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.415197 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.414987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.415197 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.415068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.415501 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.415396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.416036 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.415663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.416036 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.415962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.416204 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.416118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.416761 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.416737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.416856 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.416763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.417352 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.417333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.417423 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.417412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.417551 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.417531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.417998 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.417982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.429554 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.429536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5m6\" (UniqueName: \"kubernetes.io/projected/7b6cedb2-5f76-4edb-8a15-b87aee32ee5f-kube-api-access-7p5m6\") pod \"prometheus-k8s-0\" (UID: \"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.509595 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.509561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:49:23.790893 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:23.790856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 02:49:23.794663 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:49:23.794639    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6cedb2_5f76_4edb_8a15_b87aee32ee5f.slice/crio-8479aa5d7cac0ec3b4b44bb5fdf774f9f68908a4812610d39c63f04dd4164232 WatchSource:0}: Error finding container 8479aa5d7cac0ec3b4b44bb5fdf774f9f68908a4812610d39c63f04dd4164232: Status 404 returned error can't find the container with id 8479aa5d7cac0ec3b4b44bb5fdf774f9f68908a4812610d39c63f04dd4164232
Sep 10 02:49:24.130005 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:24.129975    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b6cedb2-5f76-4edb-8a15-b87aee32ee5f" containerID="567a41227d1bc8baa20c9128e54fac283acaa4cd5d8c0fb3d1f4c4d78541ecfa" exitCode=0
Sep 10 02:49:24.130328 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:24.130038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerDied","Data":"567a41227d1bc8baa20c9128e54fac283acaa4cd5d8c0fb3d1f4c4d78541ecfa"}
Sep 10 02:49:24.130328 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:24.130086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerStarted","Data":"8479aa5d7cac0ec3b4b44bb5fdf774f9f68908a4812610d39c63f04dd4164232"}
Sep 10 02:49:25.137067 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:25.137032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerStarted","Data":"8c747d360b9a53a2960c60bb3bbb49d9d1d232c0e133525b42332ac04d4deb65"}
Sep 10 02:49:25.137479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:25.137098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerStarted","Data":"0fe05eb9747439d827698fb9fe1c151b0087594ef9a45854ce2a0a439f04a0e2"}
Sep 10 02:49:25.137479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:25.137159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerStarted","Data":"eaa88ec7f130d4268b350a947d6a3660bf8086ae6552e47ea942de46b10b68f5"}
Sep 10 02:49:25.137479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:25.137212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerStarted","Data":"93646bd48f9e0aae4ea6339e16914036ea7ce2504cb38a3eb77cc285a18ef0cc"}
Sep 10 02:49:25.137479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:25.137238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerStarted","Data":"5eb9640cbf366ed9641b84c49f922cd59b3bd030b05f8d0d765f199f0fb9de16"}
Sep 10 02:49:25.137479 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:25.137260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b6cedb2-5f76-4edb-8a15-b87aee32ee5f","Type":"ContainerStarted","Data":"d7d73cfca4601a2ecdb91818ee2492c1ca1e41381cc28cb50791ed964af71958"}
Sep 10 02:49:25.167002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:25.166953    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.144148311 podStartE2EDuration="2.166937698s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.789387ms" isStatefulPod=false podCreationTimestamp="2026-09-10 02:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:49:25.165093556 +0000 UTC m=+162.314993266" watchObservedRunningTime="2026-09-10 02:49:25.166937698 +0000 UTC m=+162.316837396"
Sep 10 02:49:28.510380 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:49:28.510352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:50:23.509987 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:50:23.509952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:50:23.525021 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:50:23.524994    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:50:24.386238 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:50:24.386208    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 02:51:03.203338 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.203304    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"]
Sep 10 02:51:03.206685 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.206666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.209769 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.209751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 02:51:03.211132 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.211112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zg64k\""
Sep 10 02:51:03.211205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.211112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 02:51:03.222934 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.222901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"]
Sep 10 02:51:03.297870 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.297825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696s2\" (UniqueName: \"kubernetes.io/projected/e8d71569-4e14-494d-9924-7b5d10a89eff-kube-api-access-696s2\") pod \"cert-manager-cainjector-559cdd8c87-5mttp\" (UID: \"e8d71569-4e14-494d-9924-7b5d10a89eff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.298051 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.297888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d71569-4e14-494d-9924-7b5d10a89eff-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5mttp\" (UID: \"e8d71569-4e14-494d-9924-7b5d10a89eff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.398753 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.398720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696s2\" (UniqueName: \"kubernetes.io/projected/e8d71569-4e14-494d-9924-7b5d10a89eff-kube-api-access-696s2\") pod \"cert-manager-cainjector-559cdd8c87-5mttp\" (UID: \"e8d71569-4e14-494d-9924-7b5d10a89eff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.398918 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.398788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d71569-4e14-494d-9924-7b5d10a89eff-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5mttp\" (UID: \"e8d71569-4e14-494d-9924-7b5d10a89eff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.411624 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.411601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d71569-4e14-494d-9924-7b5d10a89eff-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5mttp\" (UID: \"e8d71569-4e14-494d-9924-7b5d10a89eff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.411789 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.411771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-696s2\" (UniqueName: \"kubernetes.io/projected/e8d71569-4e14-494d-9924-7b5d10a89eff-kube-api-access-696s2\") pod \"cert-manager-cainjector-559cdd8c87-5mttp\" (UID: \"e8d71569-4e14-494d-9924-7b5d10a89eff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.529265 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.529194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"
Sep 10 02:51:03.803538 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:03.803509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5mttp"]
Sep 10 02:51:03.805091 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:51:03.805062    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8d71569_4e14_494d_9924_7b5d10a89eff.slice/crio-76114352f4deb75d36a26e79deee56c87e62fab7de94142ffb9660e3de74b71c WatchSource:0}: Error finding container 76114352f4deb75d36a26e79deee56c87e62fab7de94142ffb9660e3de74b71c: Status 404 returned error can't find the container with id 76114352f4deb75d36a26e79deee56c87e62fab7de94142ffb9660e3de74b71c
Sep 10 02:51:04.519521 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:04.519487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp" event={"ID":"e8d71569-4e14-494d-9924-7b5d10a89eff","Type":"ContainerStarted","Data":"76114352f4deb75d36a26e79deee56c87e62fab7de94142ffb9660e3de74b71c"}
Sep 10 02:51:07.534425 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:07.534388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp" event={"ID":"e8d71569-4e14-494d-9924-7b5d10a89eff","Type":"ContainerStarted","Data":"f5bea85f5a056b6a078a85762761c6b823b350c2309ceb7054b1267f6a809462"}
Sep 10 02:51:07.552155 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:07.552093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5mttp" podStartSLOduration=1.382381576 podStartE2EDuration="4.552076737s" totalImagesPullingTime="3.169695161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:07.551074495 +0000 UTC m=+264.700974195" watchObservedRunningTime="2026-09-10 02:51:07.552076737 +0000 UTC m=+264.701976434"
Sep 10 02:51:34.127435 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.127393    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"]
Sep 10 02:51:34.131022 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.131002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.134212 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.134184    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 02:51:34.134372 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.134249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 02:51:34.135720 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.135697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 02:51:34.135852 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.135732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gkvsz\""
Sep 10 02:51:34.135852 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.135706    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 02:51:34.135852 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.135785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 02:51:34.140937 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.140906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"]
Sep 10 02:51:34.264388 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.264341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0235f993-41e9-4711-bc40-4a001a538495-manager-config\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.264388 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.264387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0235f993-41e9-4711-bc40-4a001a538495-cert\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.264609 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.264487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0235f993-41e9-4711-bc40-4a001a538495-metrics-cert\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.264609 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.264519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvx42\" (UniqueName: \"kubernetes.io/projected/0235f993-41e9-4711-bc40-4a001a538495-kube-api-access-cvx42\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.365848 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.365809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0235f993-41e9-4711-bc40-4a001a538495-metrics-cert\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.365848 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.365847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvx42\" (UniqueName: \"kubernetes.io/projected/0235f993-41e9-4711-bc40-4a001a538495-kube-api-access-cvx42\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.366078 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.365918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0235f993-41e9-4711-bc40-4a001a538495-manager-config\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.366078 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.365937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0235f993-41e9-4711-bc40-4a001a538495-cert\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.366801 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.366765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0235f993-41e9-4711-bc40-4a001a538495-manager-config\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.368886 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.368852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0235f993-41e9-4711-bc40-4a001a538495-metrics-cert\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.369070 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.369017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0235f993-41e9-4711-bc40-4a001a538495-cert\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.379476 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.379385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvx42\" (UniqueName: \"kubernetes.io/projected/0235f993-41e9-4711-bc40-4a001a538495-kube-api-access-cvx42\") pod \"lws-controller-manager-6cd97c998c-h65pb\" (UID: \"0235f993-41e9-4711-bc40-4a001a538495\") " pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.441664 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.441621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:34.727223 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:34.727109    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"]
Sep 10 02:51:34.730990 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:51:34.730956    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0235f993_41e9_4711_bc40_4a001a538495.slice/crio-5980bec3cd63afd87904abf8f08681429121de8a4387e03cea7b3b8fbc8706a7 WatchSource:0}: Error finding container 5980bec3cd63afd87904abf8f08681429121de8a4387e03cea7b3b8fbc8706a7: Status 404 returned error can't find the container with id 5980bec3cd63afd87904abf8f08681429121de8a4387e03cea7b3b8fbc8706a7
Sep 10 02:51:35.647647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:35.647604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb" event={"ID":"0235f993-41e9-4711-bc40-4a001a538495","Type":"ContainerStarted","Data":"5980bec3cd63afd87904abf8f08681429121de8a4387e03cea7b3b8fbc8706a7"}
Sep 10 02:51:37.656778 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:37.656737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb" event={"ID":"0235f993-41e9-4711-bc40-4a001a538495","Type":"ContainerStarted","Data":"ddd78bacd8741214b84df7075a25788d5d2ed9180ea7944b807eda83b905a182"}
Sep 10 02:51:37.657157 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:37.656880    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:37.674777 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:37.674721    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb" podStartSLOduration=1.119733905 podStartE2EDuration="3.674703613s" totalImagesPullingTime="2.554969708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:37.673667394 +0000 UTC m=+294.823567091" watchObservedRunningTime="2026-09-10 02:51:37.674703613 +0000 UTC m=+294.824603310"
Sep 10 02:51:43.283073 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:43.283039    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 02:51:47.057364 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.057323    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"]
Sep 10 02:51:47.059724 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.059703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.067253 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.067232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 02:51:47.067864 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.067846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 02:51:47.067979 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.067904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 02:51:47.067979 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.067909    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fcm22\""
Sep 10 02:51:47.068134 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.068053    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 02:51:47.093338 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.093310    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"]
Sep 10 02:51:47.185593 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.185556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a037c5-f39c-4a10-8113-a3fd352c51d8-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.185593 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.185593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5a037c5-f39c-4a10-8113-a3fd352c51d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.185794 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.185617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58gf\" (UniqueName: \"kubernetes.io/projected/c5a037c5-f39c-4a10-8113-a3fd352c51d8-kube-api-access-q58gf\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.185794 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.185750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5a037c5-f39c-4a10-8113-a3fd352c51d8-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.185794 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.185776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5a037c5-f39c-4a10-8113-a3fd352c51d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.286706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.286671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5a037c5-f39c-4a10-8113-a3fd352c51d8-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.286706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.286704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5a037c5-f39c-4a10-8113-a3fd352c51d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.286955 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.286734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a037c5-f39c-4a10-8113-a3fd352c51d8-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.286955 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.286751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5a037c5-f39c-4a10-8113-a3fd352c51d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.286955 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.286779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58gf\" (UniqueName: \"kubernetes.io/projected/c5a037c5-f39c-4a10-8113-a3fd352c51d8-kube-api-access-q58gf\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.287125 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.287101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5a037c5-f39c-4a10-8113-a3fd352c51d8-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.288962 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.288934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a037c5-f39c-4a10-8113-a3fd352c51d8-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.289269 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.289250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5a037c5-f39c-4a10-8113-a3fd352c51d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.289395 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.289290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5a037c5-f39c-4a10-8113-a3fd352c51d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.295138 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.295117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58gf\" (UniqueName: \"kubernetes.io/projected/c5a037c5-f39c-4a10-8113-a3fd352c51d8-kube-api-access-q58gf\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-bk7nj\" (UID: \"c5a037c5-f39c-4a10-8113-a3fd352c51d8\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.370287 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.370258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:47.659819 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.659794    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"]
Sep 10 02:51:47.662231 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:51:47.662191    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a037c5_f39c_4a10_8113_a3fd352c51d8.slice/crio-c818eb4ed22fb024cad7fc65d74720425a24f8679a77d892ac82a4c4051365b5 WatchSource:0}: Error finding container c818eb4ed22fb024cad7fc65d74720425a24f8679a77d892ac82a4c4051365b5: Status 404 returned error can't find the container with id c818eb4ed22fb024cad7fc65d74720425a24f8679a77d892ac82a4c4051365b5
Sep 10 02:51:47.664178 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.664145    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 02:51:47.697547 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:47.697518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj" event={"ID":"c5a037c5-f39c-4a10-8113-a3fd352c51d8","Type":"ContainerStarted","Data":"c818eb4ed22fb024cad7fc65d74720425a24f8679a77d892ac82a4c4051365b5"}
Sep 10 02:51:48.663209 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:48.663179    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cd97c998c-h65pb"
Sep 10 02:51:50.716213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:50.716157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj" event={"ID":"c5a037c5-f39c-4a10-8113-a3fd352c51d8","Type":"ContainerStarted","Data":"cefd90e69aecefb5b556214453c93bf2bf0b9e52c3bd6a5c99c13e83ecc7636e"}
Sep 10 02:51:52.726641 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:52.726605    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5a037c5-f39c-4a10-8113-a3fd352c51d8" containerID="cefd90e69aecefb5b556214453c93bf2bf0b9e52c3bd6a5c99c13e83ecc7636e" exitCode=0
Sep 10 02:51:52.726641 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:52.726644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj" event={"ID":"c5a037c5-f39c-4a10-8113-a3fd352c51d8","Type":"ContainerDied","Data":"cefd90e69aecefb5b556214453c93bf2bf0b9e52c3bd6a5c99c13e83ecc7636e"}
Sep 10 02:51:53.738084 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:53.738050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj" event={"ID":"c5a037c5-f39c-4a10-8113-a3fd352c51d8","Type":"ContainerStarted","Data":"d570d860c65cbb9c8e134a772bc373a9b992dfa3135f3e0b3b1db68a79e8898b"}
Sep 10 02:51:53.738466 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:53.738302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:51:53.761200 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:51:53.761139    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj" podStartSLOduration=2.479041592 podStartE2EDuration="6.76112614s" totalImagesPullingTime="3.208064008s" totalInitContainerRuntime="1.07402054s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:53.760298955 +0000 UTC m=+310.910198654" watchObservedRunningTime="2026-09-10 02:51:53.76112614 +0000 UTC m=+310.911025837"
Sep 10 02:52:04.745315 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:04.745286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-bk7nj"
Sep 10 02:52:06.723794 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.723761    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"]
Sep 10 02:52:06.727118 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.727101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.730333 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.730313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 02:52:06.730436 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.730317    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 02:52:06.730719 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.730703    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 02:52:06.731762 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.731744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fb2p4\""
Sep 10 02:52:06.731850 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.731763    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 02:52:06.738954 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.738928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"]
Sep 10 02:52:06.827037 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.827001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630a69ea-e5b1-473c-b15e-a21790a98a34-tls-certs\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.827217 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.827043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2kdh\" (UniqueName: \"kubernetes.io/projected/630a69ea-e5b1-473c-b15e-a21790a98a34-kube-api-access-l2kdh\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.827217 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.827105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630a69ea-e5b1-473c-b15e-a21790a98a34-tmp\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.927769 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.927669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630a69ea-e5b1-473c-b15e-a21790a98a34-tls-certs\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.927917 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.927775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2kdh\" (UniqueName: \"kubernetes.io/projected/630a69ea-e5b1-473c-b15e-a21790a98a34-kube-api-access-l2kdh\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.927917 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.927816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630a69ea-e5b1-473c-b15e-a21790a98a34-tmp\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.930010 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.929987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630a69ea-e5b1-473c-b15e-a21790a98a34-tmp\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.930117 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.930061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630a69ea-e5b1-473c-b15e-a21790a98a34-tls-certs\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:06.935493 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:06.935473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2kdh\" (UniqueName: \"kubernetes.io/projected/630a69ea-e5b1-473c-b15e-a21790a98a34-kube-api-access-l2kdh\") pod \"kube-auth-proxy-759bbcf6c6-mpp57\" (UID: \"630a69ea-e5b1-473c-b15e-a21790a98a34\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:07.037295 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:07.037236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"
Sep 10 02:52:07.323131 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:52:07.323104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630a69ea_e5b1_473c_b15e_a21790a98a34.slice/crio-c188d6faba3f854cc46ef49984fe4a84ac3416ae732bb6778c1011b44b522125 WatchSource:0}: Error finding container c188d6faba3f854cc46ef49984fe4a84ac3416ae732bb6778c1011b44b522125: Status 404 returned error can't find the container with id c188d6faba3f854cc46ef49984fe4a84ac3416ae732bb6778c1011b44b522125
Sep 10 02:52:07.325002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:07.324984    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57"]
Sep 10 02:52:07.798907 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:07.798869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57" event={"ID":"630a69ea-e5b1-473c-b15e-a21790a98a34","Type":"ContainerStarted","Data":"c188d6faba3f854cc46ef49984fe4a84ac3416ae732bb6778c1011b44b522125"}
Sep 10 02:52:10.815715 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:10.815675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57" event={"ID":"630a69ea-e5b1-473c-b15e-a21790a98a34","Type":"ContainerStarted","Data":"66a08b12958f63a77e58af6265a0367cc4b06ffe689cceea66d39e9c5138dfc0"}
Sep 10 02:52:10.831885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:52:10.831832    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-mpp57" podStartSLOduration=2.1441887299999998 podStartE2EDuration="4.831816855s" totalImagesPullingTime="2.687628125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:10.830666591 +0000 UTC m=+327.980566290" watchObservedRunningTime="2026-09-10 02:52:10.831816855 +0000 UTC m=+327.981716563"
Sep 10 02:54:05.475592 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.475547    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"]
Sep 10 02:54:05.479107 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.479091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:05.481946 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.481926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sp59z\""
Sep 10 02:54:05.482074 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.481924    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 02:54:05.482074 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.481924    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 02:54:05.494350 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.494323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"]
Sep 10 02:54:05.563787 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.563756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:05.563911 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.563833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ckrt\" (UniqueName: \"kubernetes.io/projected/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-kube-api-access-5ckrt\") pod \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:05.583647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.583620    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"]
Sep 10 02:54:05.583837 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:54:05.583819    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-5ckrt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp" podUID="d6d809a9-5ddd-475c-bf43-f7b9fcf97254"
Sep 10 02:54:05.665104 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.665072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:05.665263 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.665118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ckrt\" (UniqueName: \"kubernetes.io/projected/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-kube-api-access-5ckrt\") pod \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:05.665563 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.665542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:05.676341 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:05.676318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ckrt\" (UniqueName: \"kubernetes.io/projected/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-kube-api-access-5ckrt\") pod \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:06.121784 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.121707    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"]
Sep 10 02:54:06.129806 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.129777    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"]
Sep 10 02:54:06.177794 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.177764    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"]
Sep 10 02:54:06.182051 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.182026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.192739 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.192713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"]
Sep 10 02:54:06.269691 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.269660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmrq\" (UniqueName: \"kubernetes.io/projected/5171f61c-8738-4f17-b735-b2e3162fdfbc-kube-api-access-chmrq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zsznh\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.269840 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.269761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5171f61c-8738-4f17-b735-b2e3162fdfbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zsznh\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.331520 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.331495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:06.334093 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:54:06.334062    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="d6d809a9-5ddd-475c-bf43-f7b9fcf97254" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:06.337960 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.337942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:06.340097 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:54:06.340075    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="d6d809a9-5ddd-475c-bf43-f7b9fcf97254" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:06.370433 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.370409    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ckrt\" (UniqueName: \"kubernetes.io/projected/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-kube-api-access-5ckrt\") pod \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") "
Sep 10 02:54:06.370540 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.370454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-extensions-socket-volume\") pod \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\" (UID: \"d6d809a9-5ddd-475c-bf43-f7b9fcf97254\") "
Sep 10 02:54:06.370540 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.370513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmrq\" (UniqueName: \"kubernetes.io/projected/5171f61c-8738-4f17-b735-b2e3162fdfbc-kube-api-access-chmrq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zsznh\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.370651 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.370586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5171f61c-8738-4f17-b735-b2e3162fdfbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zsznh\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.370713 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.370687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d6d809a9-5ddd-475c-bf43-f7b9fcf97254" (UID: "d6d809a9-5ddd-475c-bf43-f7b9fcf97254"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:54:06.370958 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.370940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5171f61c-8738-4f17-b735-b2e3162fdfbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zsznh\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.372455 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.372411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-kube-api-access-5ckrt" (OuterVolumeSpecName: "kube-api-access-5ckrt") pod "d6d809a9-5ddd-475c-bf43-f7b9fcf97254" (UID: "d6d809a9-5ddd-475c-bf43-f7b9fcf97254"). InnerVolumeSpecName "kube-api-access-5ckrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:54:06.392623 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.392597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmrq\" (UniqueName: \"kubernetes.io/projected/5171f61c-8738-4f17-b735-b2e3162fdfbc-kube-api-access-chmrq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zsznh\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.471302 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.471277    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ckrt\" (UniqueName: \"kubernetes.io/projected/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-kube-api-access-5ckrt\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:54:06.471302 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.471299    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d809a9-5ddd-475c-bf43-f7b9fcf97254-extensions-socket-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:54:06.496270 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.496248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:06.768935 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:06.768861    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"]
Sep 10 02:54:06.771985 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:54:06.771956    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5171f61c_8738_4f17_b735_b2e3162fdfbc.slice/crio-0ce40bc261ef9f6bbb072e59f93b2a35344c497214d72035b11e402822a5d0df WatchSource:0}: Error finding container 0ce40bc261ef9f6bbb072e59f93b2a35344c497214d72035b11e402822a5d0df: Status 404 returned error can't find the container with id 0ce40bc261ef9f6bbb072e59f93b2a35344c497214d72035b11e402822a5d0df
Sep 10 02:54:07.324333 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:07.324299    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6d809a9-5ddd-475c-bf43-f7b9fcf97254" path="/var/lib/kubelet/pods/d6d809a9-5ddd-475c-bf43-f7b9fcf97254/volumes"
Sep 10 02:54:07.337220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:07.337195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh" event={"ID":"5171f61c-8738-4f17-b735-b2e3162fdfbc","Type":"ContainerStarted","Data":"0ce40bc261ef9f6bbb072e59f93b2a35344c497214d72035b11e402822a5d0df"}
Sep 10 02:54:07.337220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:07.337213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:07.341367 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:54:07.341344    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-gt8qp\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="d6d809a9-5ddd-475c-bf43-f7b9fcf97254" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gt8qp"
Sep 10 02:54:11.359492 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:11.359454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh" event={"ID":"5171f61c-8738-4f17-b735-b2e3162fdfbc","Type":"ContainerStarted","Data":"01cd95c4e0ce71629da5a3f4bbd2a5ba9e97b7ea1bdcabf5a0d6db2413157e49"}
Sep 10 02:54:11.359946 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:11.359539    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:11.383446 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:11.383392    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh" podStartSLOduration=1.546943526 podStartE2EDuration="5.383378245s" totalImagesPullingTime="3.836434719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:11.382794415 +0000 UTC m=+448.532694111" watchObservedRunningTime="2026-09-10 02:54:11.383378245 +0000 UTC m=+448.533277937"
Sep 10 02:54:18.154388 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.154351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"]
Sep 10 02:54:18.154805 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.154576    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh" podUID="5171f61c-8738-4f17-b735-b2e3162fdfbc" containerName="manager" containerID="cri-o://01cd95c4e0ce71629da5a3f4bbd2a5ba9e97b7ea1bdcabf5a0d6db2413157e49" gracePeriod=10
Sep 10 02:54:18.156244 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.156219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:18.391181 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.391133    2580 generic.go:356] "Generic (PLEG): container finished" podID="5171f61c-8738-4f17-b735-b2e3162fdfbc" containerID="01cd95c4e0ce71629da5a3f4bbd2a5ba9e97b7ea1bdcabf5a0d6db2413157e49" exitCode=0
Sep 10 02:54:18.391312 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.391200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh" event={"ID":"5171f61c-8738-4f17-b735-b2e3162fdfbc","Type":"ContainerDied","Data":"01cd95c4e0ce71629da5a3f4bbd2a5ba9e97b7ea1bdcabf5a0d6db2413157e49"}
Sep 10 02:54:18.391312 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.391258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh" event={"ID":"5171f61c-8738-4f17-b735-b2e3162fdfbc","Type":"ContainerDied","Data":"0ce40bc261ef9f6bbb072e59f93b2a35344c497214d72035b11e402822a5d0df"}
Sep 10 02:54:18.391312 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.391280    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce40bc261ef9f6bbb072e59f93b2a35344c497214d72035b11e402822a5d0df"
Sep 10 02:54:18.395976 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.395949    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:18.481610 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.481541    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5171f61c-8738-4f17-b735-b2e3162fdfbc-extensions-socket-volume\") pod \"5171f61c-8738-4f17-b735-b2e3162fdfbc\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") "
Sep 10 02:54:18.481736 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.481616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chmrq\" (UniqueName: \"kubernetes.io/projected/5171f61c-8738-4f17-b735-b2e3162fdfbc-kube-api-access-chmrq\") pod \"5171f61c-8738-4f17-b735-b2e3162fdfbc\" (UID: \"5171f61c-8738-4f17-b735-b2e3162fdfbc\") "
Sep 10 02:54:18.482061 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.482032    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5171f61c-8738-4f17-b735-b2e3162fdfbc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5171f61c-8738-4f17-b735-b2e3162fdfbc" (UID: "5171f61c-8738-4f17-b735-b2e3162fdfbc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:54:18.483733 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.483713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5171f61c-8738-4f17-b735-b2e3162fdfbc-kube-api-access-chmrq" (OuterVolumeSpecName: "kube-api-access-chmrq") pod "5171f61c-8738-4f17-b735-b2e3162fdfbc" (UID: "5171f61c-8738-4f17-b735-b2e3162fdfbc"). InnerVolumeSpecName "kube-api-access-chmrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:54:18.583294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.583264    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmrq\" (UniqueName: \"kubernetes.io/projected/5171f61c-8738-4f17-b735-b2e3162fdfbc-kube-api-access-chmrq\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:54:18.583294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:18.583287    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5171f61c-8738-4f17-b735-b2e3162fdfbc-extensions-socket-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:54:19.395268 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:19.395239    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"
Sep 10 02:54:19.433152 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:19.433120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"]
Sep 10 02:54:19.441471 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:19.441447    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zsznh"]
Sep 10 02:54:21.330251 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:21.330214    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5171f61c-8738-4f17-b735-b2e3162fdfbc" path="/var/lib/kubelet/pods/5171f61c-8738-4f17-b735-b2e3162fdfbc/volumes"
Sep 10 02:54:34.560269 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.560237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"]
Sep 10 02:54:34.560819 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.560802    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5171f61c-8738-4f17-b735-b2e3162fdfbc" containerName="manager"
Sep 10 02:54:34.560864 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.560824    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5171f61c-8738-4f17-b735-b2e3162fdfbc" containerName="manager"
Sep 10 02:54:34.560918 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.560906    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5171f61c-8738-4f17-b735-b2e3162fdfbc" containerName="manager"
Sep 10 02:54:34.564580 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.564560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.567441 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.567419    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 02:54:34.567562 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.567460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-975wn\""
Sep 10 02:54:34.573705 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.573681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"]
Sep 10 02:54:34.716439 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716439 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716650 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716650 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716650 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.716881 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.716790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvsg\" (UniqueName: \"kubernetes.io/projected/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-kube-api-access-snvsg\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817470 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817470 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817470 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvsg\" (UniqueName: \"kubernetes.io/projected/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-kube-api-access-snvsg\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817706 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.817994 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.817892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.818065 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.818020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.818122 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.818087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.818442 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.818417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.818549 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.818529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.819733 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.819714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.820256 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.820239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.826652 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.826630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.827047 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.827012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvsg\" (UniqueName: \"kubernetes.io/projected/00b5c9d7-c6ba-46b5-aef5-919ea9f1e780-kube-api-access-snvsg\") pod \"maas-default-gateway-openshift-default-5ccb968c66-7z5m6\" (UID: \"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:34.876007 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:34.875988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:35.150211 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:35.150187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"]
Sep 10 02:54:35.151940 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:54:35.151906    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b5c9d7_c6ba_46b5_aef5_919ea9f1e780.slice/crio-9061db913ed363ecdae99982a0e4f793531b01b6374222ae0956c20dcdb29030 WatchSource:0}: Error finding container 9061db913ed363ecdae99982a0e4f793531b01b6374222ae0956c20dcdb29030: Status 404 returned error can't find the container with id 9061db913ed363ecdae99982a0e4f793531b01b6374222ae0956c20dcdb29030
Sep 10 02:54:35.463110 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:35.463039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" event={"ID":"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780","Type":"ContainerStarted","Data":"9061db913ed363ecdae99982a0e4f793531b01b6374222ae0956c20dcdb29030"}
Sep 10 02:54:37.844175 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:37.844126    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:54:37.844405 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:37.844226    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:54:37.844405 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:37.844258    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:54:38.480856 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:38.480814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" event={"ID":"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780","Type":"ContainerStarted","Data":"318a1284f22e0f8bf6c76d9528ffe9c069b309f27a89b4bbc743359043238d39"}
Sep 10 02:54:38.502489 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:38.502446    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" podStartSLOduration=1.8123887490000001 podStartE2EDuration="4.50243274s" totalImagesPullingTime="2.690043991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:38.500959767 +0000 UTC m=+475.650859464" watchObservedRunningTime="2026-09-10 02:54:38.50243274 +0000 UTC m=+475.652332436"
Sep 10 02:54:38.876207 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:38.876173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:38.880673 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:38.880652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:39.485649 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:39.485621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:39.486495 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:39.486474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:54:51.027112 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.027077    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:54:51.032339 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.032321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.035456 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.035435    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pdldg\""
Sep 10 02:54:51.035627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.035483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 02:54:51.036762 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.036745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 02:54:51.036848 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.036775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 02:54:51.040622 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.040597    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:54:51.071712 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.071690    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:54:51.146482 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.146461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfwt\" (UniqueName: \"kubernetes.io/projected/e6544dd2-ffb2-4e04-8c56-d833b526073d-kube-api-access-szfwt\") pod \"limitador-limitador-84994f7ffc-t9kpg\" (UID: \"e6544dd2-ffb2-4e04-8c56-d833b526073d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.146589 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.146492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e6544dd2-ffb2-4e04-8c56-d833b526073d-config-file\") pod \"limitador-limitador-84994f7ffc-t9kpg\" (UID: \"e6544dd2-ffb2-4e04-8c56-d833b526073d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.247568 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.247538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfwt\" (UniqueName: \"kubernetes.io/projected/e6544dd2-ffb2-4e04-8c56-d833b526073d-kube-api-access-szfwt\") pod \"limitador-limitador-84994f7ffc-t9kpg\" (UID: \"e6544dd2-ffb2-4e04-8c56-d833b526073d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.247699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.247577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e6544dd2-ffb2-4e04-8c56-d833b526073d-config-file\") pod \"limitador-limitador-84994f7ffc-t9kpg\" (UID: \"e6544dd2-ffb2-4e04-8c56-d833b526073d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.248222 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.248205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e6544dd2-ffb2-4e04-8c56-d833b526073d-config-file\") pod \"limitador-limitador-84994f7ffc-t9kpg\" (UID: \"e6544dd2-ffb2-4e04-8c56-d833b526073d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.258683 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.258663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfwt\" (UniqueName: \"kubernetes.io/projected/e6544dd2-ffb2-4e04-8c56-d833b526073d-kube-api-access-szfwt\") pod \"limitador-limitador-84994f7ffc-t9kpg\" (UID: \"e6544dd2-ffb2-4e04-8c56-d833b526073d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.342553 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.342497    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:51.514894 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.514862    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-88x7k"]
Sep 10 02:54:51.517505 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.517490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k"
Sep 10 02:54:51.520781 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.520758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5stxp\""
Sep 10 02:54:51.532225 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.532206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-88x7k"]
Sep 10 02:54:51.610176 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.610115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:54:51.651086 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.651060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjfd\" (UniqueName: \"kubernetes.io/projected/472664b4-7831-4ad2-8840-1423d0ff7142-kube-api-access-pnjfd\") pod \"authorino-7bcfd8f9cd-88x7k\" (UID: \"472664b4-7831-4ad2-8840-1423d0ff7142\") " pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k"
Sep 10 02:54:51.752141 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.752114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjfd\" (UniqueName: \"kubernetes.io/projected/472664b4-7831-4ad2-8840-1423d0ff7142-kube-api-access-pnjfd\") pod \"authorino-7bcfd8f9cd-88x7k\" (UID: \"472664b4-7831-4ad2-8840-1423d0ff7142\") " pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k"
Sep 10 02:54:51.759851 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.759832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjfd\" (UniqueName: \"kubernetes.io/projected/472664b4-7831-4ad2-8840-1423d0ff7142-kube-api-access-pnjfd\") pod \"authorino-7bcfd8f9cd-88x7k\" (UID: \"472664b4-7831-4ad2-8840-1423d0ff7142\") " pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k"
Sep 10 02:54:51.827094 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:51.827073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k"
Sep 10 02:54:52.096532 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:52.096375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-88x7k"]
Sep 10 02:54:52.098959 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:54:52.098937    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod472664b4_7831_4ad2_8840_1423d0ff7142.slice/crio-52ed7bfdda137b71a81982821482438acd8de25cf96221eeff2e8aecb964df00 WatchSource:0}: Error finding container 52ed7bfdda137b71a81982821482438acd8de25cf96221eeff2e8aecb964df00: Status 404 returned error can't find the container with id 52ed7bfdda137b71a81982821482438acd8de25cf96221eeff2e8aecb964df00
Sep 10 02:54:52.550772 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:52.550731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k" event={"ID":"472664b4-7831-4ad2-8840-1423d0ff7142","Type":"ContainerStarted","Data":"52ed7bfdda137b71a81982821482438acd8de25cf96221eeff2e8aecb964df00"}
Sep 10 02:54:52.552191 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:52.552146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg" event={"ID":"e6544dd2-ffb2-4e04-8c56-d833b526073d","Type":"ContainerStarted","Data":"4a8bcfccaa5256b7eb85989116495a2439a52c08d4b33b5b65baecbbd438ac1b"}
Sep 10 02:54:53.559749 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:53.559699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg" event={"ID":"e6544dd2-ffb2-4e04-8c56-d833b526073d","Type":"ContainerStarted","Data":"db9db17b4515e41f0fac45abdd37489769f28103b680dce1059b2e3d15990310"}
Sep 10 02:54:53.560213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:53.559855    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:54:53.577307 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:53.577268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg" podStartSLOduration=1.53540478 podStartE2EDuration="2.577255615s" totalImagesPullingTime="1.041850835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:53.576558055 +0000 UTC m=+490.726457770" watchObservedRunningTime="2026-09-10 02:54:53.577255615 +0000 UTC m=+490.727155311"
Sep 10 02:54:54.394514 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:54.394483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-88x7k"]
Sep 10 02:54:54.565920 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:54.565887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k" event={"ID":"472664b4-7831-4ad2-8840-1423d0ff7142","Type":"ContainerStarted","Data":"7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9"}
Sep 10 02:54:54.595602 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:54.595552    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k" podStartSLOduration=1.9466591420000001 podStartE2EDuration="3.595536385s" totalImagesPullingTime="1.648877243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:54.593298209 +0000 UTC m=+491.743197906" watchObservedRunningTime="2026-09-10 02:54:54.595536385 +0000 UTC m=+491.745436081"
Sep 10 02:54:55.569951 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:55.569915    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k" podUID="472664b4-7831-4ad2-8840-1423d0ff7142" containerName="authorino" containerID="cri-o://7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9" gracePeriod=30
Sep 10 02:54:55.805983 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:55.805962    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k"
Sep 10 02:54:55.984744 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:55.984714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnjfd\" (UniqueName: \"kubernetes.io/projected/472664b4-7831-4ad2-8840-1423d0ff7142-kube-api-access-pnjfd\") pod \"472664b4-7831-4ad2-8840-1423d0ff7142\" (UID: \"472664b4-7831-4ad2-8840-1423d0ff7142\") "
Sep 10 02:54:55.986713 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:55.986690    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472664b4-7831-4ad2-8840-1423d0ff7142-kube-api-access-pnjfd" (OuterVolumeSpecName: "kube-api-access-pnjfd") pod "472664b4-7831-4ad2-8840-1423d0ff7142" (UID: "472664b4-7831-4ad2-8840-1423d0ff7142"). InnerVolumeSpecName "kube-api-access-pnjfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:54:56.086118 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.086085    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjfd\" (UniqueName: \"kubernetes.io/projected/472664b4-7831-4ad2-8840-1423d0ff7142-kube-api-access-pnjfd\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:54:56.575530 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.575497    2580 generic.go:356] "Generic (PLEG): container finished" podID="472664b4-7831-4ad2-8840-1423d0ff7142" containerID="7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9" exitCode=0
Sep 10 02:54:56.575967 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.575537    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k"
Sep 10 02:54:56.575967 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.575583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k" event={"ID":"472664b4-7831-4ad2-8840-1423d0ff7142","Type":"ContainerDied","Data":"7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9"}
Sep 10 02:54:56.575967 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.575624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-88x7k" event={"ID":"472664b4-7831-4ad2-8840-1423d0ff7142","Type":"ContainerDied","Data":"52ed7bfdda137b71a81982821482438acd8de25cf96221eeff2e8aecb964df00"}
Sep 10 02:54:56.575967 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.575641    2580 scope.go:117] "RemoveContainer" containerID="7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9"
Sep 10 02:54:56.584143 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.584126    2580 scope.go:117] "RemoveContainer" containerID="7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9"
Sep 10 02:54:56.584414 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:54:56.584397    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9\": container with ID starting with 7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9 not found: ID does not exist" containerID="7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9"
Sep 10 02:54:56.584473 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.584426    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9"} err="failed to get container status \"7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9\": rpc error: code = NotFound desc = could not find container \"7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9\": container with ID starting with 7951b8117236e95ede5a9fe0f65d4c485b50e04d4b6977b941024b75ef2782c9 not found: ID does not exist"
Sep 10 02:54:56.595302 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.595281    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-88x7k"]
Sep 10 02:54:56.599247 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:56.599229    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-88x7k"]
Sep 10 02:54:57.323985 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:54:57.323953    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="472664b4-7831-4ad2-8840-1423d0ff7142" path="/var/lib/kubelet/pods/472664b4-7831-4ad2-8840-1423d0ff7142/volumes"
Sep 10 02:55:04.567656 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:04.567628    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"
Sep 10 02:55:16.019987 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.019952    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-rpfq2"]
Sep 10 02:55:16.020515 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.020330    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472664b4-7831-4ad2-8840-1423d0ff7142" containerName="authorino"
Sep 10 02:55:16.020515 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.020344    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="472664b4-7831-4ad2-8840-1423d0ff7142" containerName="authorino"
Sep 10 02:55:16.020515 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.020417    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="472664b4-7831-4ad2-8840-1423d0ff7142" containerName="authorino"
Sep 10 02:55:16.025474 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.025458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rpfq2"
Sep 10 02:55:16.028663 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.028641    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5stxp\""
Sep 10 02:55:16.030588 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.030566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-rpfq2"]
Sep 10 02:55:16.075535 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.075513    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rpfq2"]
Sep 10 02:55:16.075739 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:55:16.075719    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cpfql], unattached volumes=[], failed to process volumes=[kube-api-access-cpfql]: context canceled" pod="kuadrant-system/authorino-557f876f69-rpfq2" podUID="cd25c73f-c259-412c-a67d-43e43fb758e6"
Sep 10 02:55:16.100233 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.100208    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tvkb9"]
Sep 10 02:55:16.102521 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.102507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.105367 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.105349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 02:55:16.110959 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.110939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tvkb9"]
Sep 10 02:55:16.126453 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.126433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfql\" (UniqueName: \"kubernetes.io/projected/cd25c73f-c259-412c-a67d-43e43fb758e6-kube-api-access-cpfql\") pod \"authorino-557f876f69-rpfq2\" (UID: \"cd25c73f-c259-412c-a67d-43e43fb758e6\") " pod="kuadrant-system/authorino-557f876f69-rpfq2"
Sep 10 02:55:16.126541 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.126497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b616228b-95dd-459a-8acc-c41cd1e0da21-tls-cert\") pod \"authorino-6d4dfc649f-tvkb9\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") " pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.126541 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.126531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcnw8\" (UniqueName: \"kubernetes.io/projected/b616228b-95dd-459a-8acc-c41cd1e0da21-kube-api-access-lcnw8\") pod \"authorino-6d4dfc649f-tvkb9\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") " pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.227580 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.227554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b616228b-95dd-459a-8acc-c41cd1e0da21-tls-cert\") pod \"authorino-6d4dfc649f-tvkb9\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") " pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.227682 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.227585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcnw8\" (UniqueName: \"kubernetes.io/projected/b616228b-95dd-459a-8acc-c41cd1e0da21-kube-api-access-lcnw8\") pod \"authorino-6d4dfc649f-tvkb9\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") " pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.227682 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.227616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfql\" (UniqueName: \"kubernetes.io/projected/cd25c73f-c259-412c-a67d-43e43fb758e6-kube-api-access-cpfql\") pod \"authorino-557f876f69-rpfq2\" (UID: \"cd25c73f-c259-412c-a67d-43e43fb758e6\") " pod="kuadrant-system/authorino-557f876f69-rpfq2"
Sep 10 02:55:16.229849 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.229833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b616228b-95dd-459a-8acc-c41cd1e0da21-tls-cert\") pod \"authorino-6d4dfc649f-tvkb9\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") " pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.235578 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.235548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfql\" (UniqueName: \"kubernetes.io/projected/cd25c73f-c259-412c-a67d-43e43fb758e6-kube-api-access-cpfql\") pod \"authorino-557f876f69-rpfq2\" (UID: \"cd25c73f-c259-412c-a67d-43e43fb758e6\") " pod="kuadrant-system/authorino-557f876f69-rpfq2"
Sep 10 02:55:16.236180 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.236142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcnw8\" (UniqueName: \"kubernetes.io/projected/b616228b-95dd-459a-8acc-c41cd1e0da21-kube-api-access-lcnw8\") pod \"authorino-6d4dfc649f-tvkb9\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") " pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.298252 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.298198    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tvkb9"]
Sep 10 02:55:16.298438 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.298426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:16.332796 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.332765    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c4bcdcbd7-5tndx"]
Sep 10 02:55:16.335496 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.335474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.345904 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.345878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4bcdcbd7-5tndx"]
Sep 10 02:55:16.428859 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.428824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0de4cf2a-6164-4f5b-9d09-3467b453de86-tls-cert\") pod \"authorino-6c4bcdcbd7-5tndx\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") " pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.428996 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.428880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgt2p\" (UniqueName: \"kubernetes.io/projected/0de4cf2a-6164-4f5b-9d09-3467b453de86-kube-api-access-pgt2p\") pod \"authorino-6c4bcdcbd7-5tndx\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") " pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.529797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.529770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0de4cf2a-6164-4f5b-9d09-3467b453de86-tls-cert\") pod \"authorino-6c4bcdcbd7-5tndx\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") " pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.529912 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.529815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgt2p\" (UniqueName: \"kubernetes.io/projected/0de4cf2a-6164-4f5b-9d09-3467b453de86-kube-api-access-pgt2p\") pod \"authorino-6c4bcdcbd7-5tndx\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") " pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.532062 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.532044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0de4cf2a-6164-4f5b-9d09-3467b453de86-tls-cert\") pod \"authorino-6c4bcdcbd7-5tndx\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") " pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.538064 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.538030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgt2p\" (UniqueName: \"kubernetes.io/projected/0de4cf2a-6164-4f5b-9d09-3467b453de86-kube-api-access-pgt2p\") pod \"authorino-6c4bcdcbd7-5tndx\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") " pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.567015 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.566996    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tvkb9"]
Sep 10 02:55:16.569392 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:55:16.569368    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb616228b_95dd_459a_8acc_c41cd1e0da21.slice/crio-14eb9e0d744a9900caa32b36bc7d8025383964aebf6f1c4d4e95473328a8745e WatchSource:0}: Error finding container 14eb9e0d744a9900caa32b36bc7d8025383964aebf6f1c4d4e95473328a8745e: Status 404 returned error can't find the container with id 14eb9e0d744a9900caa32b36bc7d8025383964aebf6f1c4d4e95473328a8745e
Sep 10 02:55:16.647692 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.647668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:55:16.670735 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.670701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9" event={"ID":"b616228b-95dd-459a-8acc-c41cd1e0da21","Type":"ContainerStarted","Data":"14eb9e0d744a9900caa32b36bc7d8025383964aebf6f1c4d4e95473328a8745e"}
Sep 10 02:55:16.670839 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.670752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rpfq2"
Sep 10 02:55:16.676983 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.676963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rpfq2"
Sep 10 02:55:16.731861 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.731837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpfql\" (UniqueName: \"kubernetes.io/projected/cd25c73f-c259-412c-a67d-43e43fb758e6-kube-api-access-cpfql\") pod \"cd25c73f-c259-412c-a67d-43e43fb758e6\" (UID: \"cd25c73f-c259-412c-a67d-43e43fb758e6\") "
Sep 10 02:55:16.733609 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.733588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd25c73f-c259-412c-a67d-43e43fb758e6-kube-api-access-cpfql" (OuterVolumeSpecName: "kube-api-access-cpfql") pod "cd25c73f-c259-412c-a67d-43e43fb758e6" (UID: "cd25c73f-c259-412c-a67d-43e43fb758e6"). InnerVolumeSpecName "kube-api-access-cpfql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:55:16.832854 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.832789    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpfql\" (UniqueName: \"kubernetes.io/projected/cd25c73f-c259-412c-a67d-43e43fb758e6-kube-api-access-cpfql\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:55:16.914598 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:16.914552    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4bcdcbd7-5tndx"]
Sep 10 02:55:16.916613 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:55:16.916593    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0de4cf2a_6164_4f5b_9d09_3467b453de86.slice/crio-b1226c31cf543375861828edcb5f41ad4d358f15c2710a1852417fc424be6fd1 WatchSource:0}: Error finding container b1226c31cf543375861828edcb5f41ad4d358f15c2710a1852417fc424be6fd1: Status 404 returned error can't find the container with id b1226c31cf543375861828edcb5f41ad4d358f15c2710a1852417fc424be6fd1
Sep 10 02:55:17.676940 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.676850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx" event={"ID":"0de4cf2a-6164-4f5b-9d09-3467b453de86","Type":"ContainerStarted","Data":"70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5"}
Sep 10 02:55:17.676940 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.676892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx" event={"ID":"0de4cf2a-6164-4f5b-9d09-3467b453de86","Type":"ContainerStarted","Data":"b1226c31cf543375861828edcb5f41ad4d358f15c2710a1852417fc424be6fd1"}
Sep 10 02:55:17.678458 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.678439    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rpfq2"
Sep 10 02:55:17.678590 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.678433    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9" podUID="b616228b-95dd-459a-8acc-c41cd1e0da21" containerName="authorino" containerID="cri-o://fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c" gracePeriod=30
Sep 10 02:55:17.678653 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.678439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9" event={"ID":"b616228b-95dd-459a-8acc-c41cd1e0da21","Type":"ContainerStarted","Data":"fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c"}
Sep 10 02:55:17.691335 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.691288    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx" podStartSLOduration=1.228993495 podStartE2EDuration="1.691271969s" totalImagesPullingTime="462.278474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:55:17.690987683 +0000 UTC m=+514.840887379" watchObservedRunningTime="2026-09-10 02:55:17.691271969 +0000 UTC m=+514.841171666"
Sep 10 02:55:17.716612 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.716582    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rpfq2"]
Sep 10 02:55:17.729307 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.729283    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-rpfq2"]
Sep 10 02:55:17.744487 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.744433    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9" podStartSLOduration=1.1960526009999999 podStartE2EDuration="1.744412947s" totalImagesPullingTime="548.360346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:55:17.74421311 +0000 UTC m=+514.894112835" watchObservedRunningTime="2026-09-10 02:55:17.744412947 +0000 UTC m=+514.894312645"
Sep 10 02:55:17.903826 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.903799    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:17.943240 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.943171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b616228b-95dd-459a-8acc-c41cd1e0da21-tls-cert\") pod \"b616228b-95dd-459a-8acc-c41cd1e0da21\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") "
Sep 10 02:55:17.943366 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.943288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcnw8\" (UniqueName: \"kubernetes.io/projected/b616228b-95dd-459a-8acc-c41cd1e0da21-kube-api-access-lcnw8\") pod \"b616228b-95dd-459a-8acc-c41cd1e0da21\" (UID: \"b616228b-95dd-459a-8acc-c41cd1e0da21\") "
Sep 10 02:55:17.945050 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.945026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b616228b-95dd-459a-8acc-c41cd1e0da21-kube-api-access-lcnw8" (OuterVolumeSpecName: "kube-api-access-lcnw8") pod "b616228b-95dd-459a-8acc-c41cd1e0da21" (UID: "b616228b-95dd-459a-8acc-c41cd1e0da21"). InnerVolumeSpecName "kube-api-access-lcnw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:55:17.952682 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:17.952658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b616228b-95dd-459a-8acc-c41cd1e0da21-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b616228b-95dd-459a-8acc-c41cd1e0da21" (UID: "b616228b-95dd-459a-8acc-c41cd1e0da21"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:55:18.044480 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.044451    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcnw8\" (UniqueName: \"kubernetes.io/projected/b616228b-95dd-459a-8acc-c41cd1e0da21-kube-api-access-lcnw8\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:55:18.044480 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.044476    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b616228b-95dd-459a-8acc-c41cd1e0da21-tls-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:55:18.684125 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.684090    2580 generic.go:356] "Generic (PLEG): container finished" podID="b616228b-95dd-459a-8acc-c41cd1e0da21" containerID="fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c" exitCode=0
Sep 10 02:55:18.684517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.684141    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9"
Sep 10 02:55:18.684517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.684189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9" event={"ID":"b616228b-95dd-459a-8acc-c41cd1e0da21","Type":"ContainerDied","Data":"fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c"}
Sep 10 02:55:18.684517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.684233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-tvkb9" event={"ID":"b616228b-95dd-459a-8acc-c41cd1e0da21","Type":"ContainerDied","Data":"14eb9e0d744a9900caa32b36bc7d8025383964aebf6f1c4d4e95473328a8745e"}
Sep 10 02:55:18.684517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.684254    2580 scope.go:117] "RemoveContainer" containerID="fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c"
Sep 10 02:55:18.692705 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.692688    2580 scope.go:117] "RemoveContainer" containerID="fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c"
Sep 10 02:55:18.692990 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:55:18.692972    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c\": container with ID starting with fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c not found: ID does not exist" containerID="fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c"
Sep 10 02:55:18.693038 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.692999    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c"} err="failed to get container status \"fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c\": rpc error: code = NotFound desc = could not find container \"fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c\": container with ID starting with fecb6d0818ba1044d0f6e003ad24420430893867492c88c36e47ee784756136c not found: ID does not exist"
Sep 10 02:55:18.704559 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.704537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tvkb9"]
Sep 10 02:55:18.708984 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:18.708965    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-tvkb9"]
Sep 10 02:55:19.324404 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:19.324367    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b616228b-95dd-459a-8acc-c41cd1e0da21" path="/var/lib/kubelet/pods/b616228b-95dd-459a-8acc-c41cd1e0da21/volumes"
Sep 10 02:55:19.324685 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:19.324673    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd25c73f-c259-412c-a67d-43e43fb758e6" path="/var/lib/kubelet/pods/cd25c73f-c259-412c-a67d-43e43fb758e6/volumes"
Sep 10 02:55:59.917418 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:55:59.917382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:56:08.914822 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:08.914792    2580 generic.go:356] "Generic (PLEG): container finished" podID="00b5c9d7-c6ba-46b5-aef5-919ea9f1e780" containerID="318a1284f22e0f8bf6c76d9528ffe9c069b309f27a89b4bbc743359043238d39" exitCode=0
Sep 10 02:56:08.915210 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:08.914860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" event={"ID":"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780","Type":"ContainerDied","Data":"318a1284f22e0f8bf6c76d9528ffe9c069b309f27a89b4bbc743359043238d39"}
Sep 10 02:56:08.915255 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:08.915239    2580 scope.go:117] "RemoveContainer" containerID="318a1284f22e0f8bf6c76d9528ffe9c069b309f27a89b4bbc743359043238d39"
Sep 10 02:56:08.916228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:08.916201    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:56:08.916283 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:08.916271    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:56:08.916321 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:08.916300    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:56:09.485863 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:09.485814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:56:09.925337 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:09.924341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" event={"ID":"00b5c9d7-c6ba-46b5-aef5-919ea9f1e780","Type":"ContainerStarted","Data":"0cdd8214c453ab7837df087619a83f555657375496aef275a1873770bf6a691a"}
Sep 10 02:56:10.876949 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:10.876909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:56:10.877733 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:10.877709    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-7z5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 10 02:56:10.877852 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:10.877756    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" podUID="00b5c9d7-c6ba-46b5-aef5-919ea9f1e780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 10 02:56:11.877382 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:11.877350    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-7z5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 10 02:56:11.877757 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:11.877408    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" podUID="00b5c9d7-c6ba-46b5-aef5-919ea9f1e780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 10 02:56:12.877385 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:12.877350    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-7z5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 10 02:56:12.877558 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:12.877409    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" podUID="00b5c9d7-c6ba-46b5-aef5-919ea9f1e780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 10 02:56:13.876862 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:13.876825    2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-7z5m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 10 02:56:13.877038 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:13.876882    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6" podUID="00b5c9d7-c6ba-46b5-aef5-919ea9f1e780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 10 02:56:14.879989 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:14.879959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:56:14.880455 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:14.880365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:56:14.881140 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:14.881119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-7z5m6"
Sep 10 02:56:31.025763 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:31.025729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:56:36.920712 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:36.920676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:56:48.414252 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:48.414217    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:56:53.610521 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:56:53.610486    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:57:02.720465 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.720426    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"]
Sep 10 02:57:02.720841 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.720798    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b616228b-95dd-459a-8acc-c41cd1e0da21" containerName="authorino"
Sep 10 02:57:02.720841 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.720810    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b616228b-95dd-459a-8acc-c41cd1e0da21" containerName="authorino"
Sep 10 02:57:02.720925 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.720911    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b616228b-95dd-459a-8acc-c41cd1e0da21" containerName="authorino"
Sep 10 02:57:02.724123 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.724107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.726834 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.726812    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 02:57:02.726834 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.726820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 02:57:02.728211 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.728193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 02:57:02.728310 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.728231    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dlt7k\""
Sep 10 02:57:02.732812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.732792    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"]
Sep 10 02:57:02.829289 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.829248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.829289 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.829290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.829498 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.829326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb158dd-cb2f-4f14-8189-c500e5ad450d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.829498 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.829342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdrc\" (UniqueName: \"kubernetes.io/projected/5cb158dd-cb2f-4f14-8189-c500e5ad450d-kube-api-access-czdrc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.829498 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.829411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.829498 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.829443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930275 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb158dd-cb2f-4f14-8189-c500e5ad450d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930437 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdrc\" (UniqueName: \"kubernetes.io/projected/5cb158dd-cb2f-4f14-8189-c500e5ad450d-kube-api-access-czdrc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930437 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930437 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930437 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930653 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930816 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930918 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.930918 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.930885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.932490 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.932469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb158dd-cb2f-4f14-8189-c500e5ad450d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.932621 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.932606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb158dd-cb2f-4f14-8189-c500e5ad450d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:02.938738 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:02.938721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdrc\" (UniqueName: \"kubernetes.io/projected/5cb158dd-cb2f-4f14-8189-c500e5ad450d-kube-api-access-czdrc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8\" (UID: \"5cb158dd-cb2f-4f14-8189-c500e5ad450d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:03.034473 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:03.034411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:03.302095 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:03.302072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"]
Sep 10 02:57:03.304244 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:57:03.304214    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb158dd_cb2f_4f14_8189_c500e5ad450d.slice/crio-395f6e34e727a20afb308d3505610d43d96eaf8a9390e67c4cb0f06a0f9bf6b7 WatchSource:0}: Error finding container 395f6e34e727a20afb308d3505610d43d96eaf8a9390e67c4cb0f06a0f9bf6b7: Status 404 returned error can't find the container with id 395f6e34e727a20afb308d3505610d43d96eaf8a9390e67c4cb0f06a0f9bf6b7
Sep 10 02:57:03.305417 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:03.305397    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 02:57:04.178363 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:04.178325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8" event={"ID":"5cb158dd-cb2f-4f14-8189-c500e5ad450d","Type":"ContainerStarted","Data":"395f6e34e727a20afb308d3505610d43d96eaf8a9390e67c4cb0f06a0f9bf6b7"}
Sep 10 02:57:04.209035 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:04.209006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:57:05.185323 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:05.185152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8" event={"ID":"5cb158dd-cb2f-4f14-8189-c500e5ad450d","Type":"ContainerStarted","Data":"43dbbcbaaef2a86343efae05004829a9cf38b903809cbf9921c5207a6dce68fc"}
Sep 10 02:57:05.205753 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:05.205690    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8" podStartSLOduration=2.2781209430000002 podStartE2EDuration="3.205674027s" totalImagesPullingTime="927.553084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:57:05.204020877 +0000 UTC m=+622.353920575" watchObservedRunningTime="2026-09-10 02:57:05.205674027 +0000 UTC m=+622.355573724"
Sep 10 02:57:11.831866 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:11.831782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:57:13.034917 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:13.034874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:13.035403 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:13.034982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:13.049115 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:13.049086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:13.248556 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:13.248528    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8"
Sep 10 02:57:43.028262 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.028230    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"]
Sep 10 02:57:43.031715 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.031695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.034878 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.034855    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 02:57:43.041911 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.041889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"]
Sep 10 02:57:43.160546 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.160517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wns\" (UniqueName: \"kubernetes.io/projected/a586ba56-1ab2-46c2-abab-5a2316319eae-kube-api-access-96wns\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.160709 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.160552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.160709 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.160580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.160709 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.160665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.160821 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.160716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a586ba56-1ab2-46c2-abab-5a2316319eae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.160821 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.160757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261152 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a586ba56-1ab2-46c2-abab-5a2316319eae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261326 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261326 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wns\" (UniqueName: \"kubernetes.io/projected/a586ba56-1ab2-46c2-abab-5a2316319eae-kube-api-access-96wns\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261326 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261326 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261326 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261588 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261626 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.261670 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.261651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.263460 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.263440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a586ba56-1ab2-46c2-abab-5a2316319eae-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.264271 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.264255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 02:57:43.270968 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.270940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wns\" (UniqueName: \"kubernetes.io/projected/a586ba56-1ab2-46c2-abab-5a2316319eae-kube-api-access-96wns\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.273562 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.273542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a586ba56-1ab2-46c2-abab-5a2316319eae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g65gp\" (UID: \"a586ba56-1ab2-46c2-abab-5a2316319eae\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.342476 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.342420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:43.616665 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:43.616642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"]
Sep 10 02:57:43.618488 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:57:43.618464    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda586ba56_1ab2_46c2_abab_5a2316319eae.slice/crio-756c316e0cd1e200da14be8097aeaa8f3b57037ad98ab579898496e55c0d6004 WatchSource:0}: Error finding container 756c316e0cd1e200da14be8097aeaa8f3b57037ad98ab579898496e55c0d6004: Status 404 returned error can't find the container with id 756c316e0cd1e200da14be8097aeaa8f3b57037ad98ab579898496e55c0d6004
Sep 10 02:57:44.380993 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:44.380957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp" event={"ID":"a586ba56-1ab2-46c2-abab-5a2316319eae","Type":"ContainerStarted","Data":"e55226e3750edc15ab8a1db76a7843f3a48c3396943bceb14e152bf606105fff"}
Sep 10 02:57:44.380993 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:44.380999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp" event={"ID":"a586ba56-1ab2-46c2-abab-5a2316319eae","Type":"ContainerStarted","Data":"756c316e0cd1e200da14be8097aeaa8f3b57037ad98ab579898496e55c0d6004"}
Sep 10 02:57:44.397650 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:44.397607    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp" podStartSLOduration=1.228900369 podStartE2EDuration="1.397592952s" totalImagesPullingTime="168.692583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:57:44.39692704 +0000 UTC m=+661.546826739" watchObservedRunningTime="2026-09-10 02:57:44.397592952 +0000 UTC m=+661.547492649"
Sep 10 02:57:45.210563 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:45.210522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:57:53.343484 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:53.343456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:53.343484 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:53.343488    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:53.355562 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:53.355538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:57:53.434712 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:57:53.434681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g65gp"
Sep 10 02:58:35.546314 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.546280    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4bcdcbd7-5tndx"]
Sep 10 02:58:35.546876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.546457    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx" podUID="0de4cf2a-6164-4f5b-9d09-3467b453de86" containerName="authorino" containerID="cri-o://70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5" gracePeriod=30
Sep 10 02:58:35.779194 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.779154    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:58:35.884465 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.884437    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgt2p\" (UniqueName: \"kubernetes.io/projected/0de4cf2a-6164-4f5b-9d09-3467b453de86-kube-api-access-pgt2p\") pod \"0de4cf2a-6164-4f5b-9d09-3467b453de86\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") "
Sep 10 02:58:35.884601 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.884551    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0de4cf2a-6164-4f5b-9d09-3467b453de86-tls-cert\") pod \"0de4cf2a-6164-4f5b-9d09-3467b453de86\" (UID: \"0de4cf2a-6164-4f5b-9d09-3467b453de86\") "
Sep 10 02:58:35.886498 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.886461    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de4cf2a-6164-4f5b-9d09-3467b453de86-kube-api-access-pgt2p" (OuterVolumeSpecName: "kube-api-access-pgt2p") pod "0de4cf2a-6164-4f5b-9d09-3467b453de86" (UID: "0de4cf2a-6164-4f5b-9d09-3467b453de86"). InnerVolumeSpecName "kube-api-access-pgt2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:58:35.893736 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.893712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de4cf2a-6164-4f5b-9d09-3467b453de86-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0de4cf2a-6164-4f5b-9d09-3467b453de86" (UID: "0de4cf2a-6164-4f5b-9d09-3467b453de86"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:58:35.986199 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.986118    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0de4cf2a-6164-4f5b-9d09-3467b453de86-tls-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:58:35.986199 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:35.986145    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgt2p\" (UniqueName: \"kubernetes.io/projected/0de4cf2a-6164-4f5b-9d09-3467b453de86-kube-api-access-pgt2p\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:58:36.627658 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.627622    2580 generic.go:356] "Generic (PLEG): container finished" podID="0de4cf2a-6164-4f5b-9d09-3467b453de86" containerID="70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5" exitCode=0
Sep 10 02:58:36.628076 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.627673    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx"
Sep 10 02:58:36.628076 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.627710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx" event={"ID":"0de4cf2a-6164-4f5b-9d09-3467b453de86","Type":"ContainerDied","Data":"70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5"}
Sep 10 02:58:36.628076 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.627752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4bcdcbd7-5tndx" event={"ID":"0de4cf2a-6164-4f5b-9d09-3467b453de86","Type":"ContainerDied","Data":"b1226c31cf543375861828edcb5f41ad4d358f15c2710a1852417fc424be6fd1"}
Sep 10 02:58:36.628076 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.627771    2580 scope.go:117] "RemoveContainer" containerID="70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5"
Sep 10 02:58:36.636648 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.636633    2580 scope.go:117] "RemoveContainer" containerID="70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5"
Sep 10 02:58:36.636903 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:58:36.636883    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5\": container with ID starting with 70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5 not found: ID does not exist" containerID="70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5"
Sep 10 02:58:36.636982 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.636913    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5"} err="failed to get container status \"70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5\": rpc error: code = NotFound desc = could not find container \"70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5\": container with ID starting with 70f8d8336772066e4cb79edd0a6eda5e790fac3402890d8b0224eecf77956ac5 not found: ID does not exist"
Sep 10 02:58:36.647982 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.647960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4bcdcbd7-5tndx"]
Sep 10 02:58:36.655727 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:36.655708    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c4bcdcbd7-5tndx"]
Sep 10 02:58:37.324287 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:37.324247    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de4cf2a-6164-4f5b-9d09-3467b453de86" path="/var/lib/kubelet/pods/0de4cf2a-6164-4f5b-9d09-3467b453de86/volumes"
Sep 10 02:58:53.756103 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.756064    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"]
Sep 10 02:58:53.756696 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.756678    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de4cf2a-6164-4f5b-9d09-3467b453de86" containerName="authorino"
Sep 10 02:58:53.756753 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.756698    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de4cf2a-6164-4f5b-9d09-3467b453de86" containerName="authorino"
Sep 10 02:58:53.756795 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.756785    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de4cf2a-6164-4f5b-9d09-3467b453de86" containerName="authorino"
Sep 10 02:58:53.761347 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.761324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.764228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.764203    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e6e6b8-openshift-default-dockercfg-fcjlt\""
Sep 10 02:58:53.773842 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.773815    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"]
Sep 10 02:58:53.841507 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-credential-socket\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841653 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-certs\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841653 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-data\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841742 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61779f37-235f-4b0d-b030-8c4c125dac42-istio-podinfo\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841742 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61779f37-235f-4b0d-b030-8c4c125dac42-istiod-ca-cert\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841827 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-istio-token\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841827 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-socket\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841827 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94xs\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-kube-api-access-b94xs\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.841933 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.841838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-envoy\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.900515 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.900475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"]
Sep 10 02:58:53.904401 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.904378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.907485 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.907465    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-097e16-openshift-default-dockercfg-fj7f7\""
Sep 10 02:58:53.917133 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.917111    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"]
Sep 10 02:58:53.942636 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-data\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.942750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-socket\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.942750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-podinfo\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.942750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-token\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.942750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-envoy\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.942750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-credential-socket\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.942994 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61779f37-235f-4b0d-b030-8c4c125dac42-istiod-ca-cert\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943048 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-data\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943098 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.942789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-envoy\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.943148 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-credential-socket\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943148 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-credential-socket\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.943277 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61779f37-235f-4b0d-b030-8c4c125dac42-istio-podinfo\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943277 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-certs\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943277 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j98q\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-kube-api-access-9j98q\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.943426 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-istio-token\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943426 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-certs\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.943426 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-socket\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943426 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94xs\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-kube-api-access-b94xs\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943426 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istiod-ca-cert\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.943669 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-data\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:53.943669 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-certs\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943669 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61779f37-235f-4b0d-b030-8c4c125dac42-istiod-ca-cert\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.943819 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.943718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-socket\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.944925 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.944908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-envoy\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.945963 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.945946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61779f37-235f-4b0d-b030-8c4c125dac42-istio-podinfo\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.952286 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.952268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94xs\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-kube-api-access-b94xs\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:53.952692 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:53.952671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-istio-token\") pod \"e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") " pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:54.044109 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-socket\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044109 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-podinfo\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044109 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-token\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-envoy\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-credential-socket\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j98q\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-kube-api-access-9j98q\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-certs\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istiod-ca-cert\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044434 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-data\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044720 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-socket\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044720 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-data\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044822 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-certs\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.044951 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.044909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-credential-socket\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.045220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.045196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istiod-ca-cert\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.046793 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.046775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-envoy\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.047221 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.047203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-podinfo\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.052279 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.052253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j98q\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-kube-api-access-9j98q\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.052601 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.052578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-token\") pod \"e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") " pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.072587 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.072556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:54.216897 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.216844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:54.367597 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.367571    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"]
Sep 10 02:58:54.368806 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:58:54.368781    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61779f37_235f_4b0d_b030_8c4c125dac42.slice/crio-ce74fd34ddcbbcdc74ecb2aa58fd3e983cb95edc615b0f74e5d63b09ea7786e0 WatchSource:0}: Error finding container ce74fd34ddcbbcdc74ecb2aa58fd3e983cb95edc615b0f74e5d63b09ea7786e0: Status 404 returned error can't find the container with id ce74fd34ddcbbcdc74ecb2aa58fd3e983cb95edc615b0f74e5d63b09ea7786e0
Sep 10 02:58:54.371041 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.371001    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:58:54.371127 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.371080    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:58:54.371127 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.371116    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:58:54.500090 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.500063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"]
Sep 10 02:58:54.502526 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:58:54.502496    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ec6504_208f_4bda_a4d8_2ae1a03203a0.slice/crio-a0a3e7d5684b9232defa934dafd4986abd0358fb701a40b76d19db6e726b64bb WatchSource:0}: Error finding container a0a3e7d5684b9232defa934dafd4986abd0358fb701a40b76d19db6e726b64bb: Status 404 returned error can't find the container with id a0a3e7d5684b9232defa934dafd4986abd0358fb701a40b76d19db6e726b64bb
Sep 10 02:58:54.505311 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.505273    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:58:54.505418 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.505344    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:58:54.505418 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.505392    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:58:54.718926 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.718834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" event={"ID":"96ec6504-208f-4bda-a4d8-2ae1a03203a0","Type":"ContainerStarted","Data":"7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15"}
Sep 10 02:58:54.718926 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.718892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" event={"ID":"96ec6504-208f-4bda-a4d8-2ae1a03203a0","Type":"ContainerStarted","Data":"a0a3e7d5684b9232defa934dafd4986abd0358fb701a40b76d19db6e726b64bb"}
Sep 10 02:58:54.720594 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.720565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" event={"ID":"61779f37-235f-4b0d-b030-8c4c125dac42","Type":"ContainerStarted","Data":"d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37"}
Sep 10 02:58:54.720710 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.720616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" event={"ID":"61779f37-235f-4b0d-b030-8c4c125dac42","Type":"ContainerStarted","Data":"ce74fd34ddcbbcdc74ecb2aa58fd3e983cb95edc615b0f74e5d63b09ea7786e0"}
Sep 10 02:58:54.739735 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.739688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podStartSLOduration=1.7396728559999999 podStartE2EDuration="1.739672856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:54.738837422 +0000 UTC m=+731.888737120" watchObservedRunningTime="2026-09-10 02:58:54.739672856 +0000 UTC m=+731.889572560"
Sep 10 02:58:54.759366 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:54.759318    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podStartSLOduration=1.759302184 podStartE2EDuration="1.759302184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:54.757457376 +0000 UTC m=+731.907357073" watchObservedRunningTime="2026-09-10 02:58:54.759302184 +0000 UTC m=+731.909201879"
Sep 10 02:58:55.073627 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.073548    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:55.078663 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.078636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:55.217301 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.217268    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:55.221917 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.221892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:55.595239 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.595201    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8024-cffd97"]
Sep 10 02:58:55.598890 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.598873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:58:55.601716 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.601692    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mtn2b\""
Sep 10 02:58:55.605633 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.605608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8024-cffd97"]
Sep 10 02:58:55.660056 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.660024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmmd\" (UniqueName: \"kubernetes.io/projected/5bba9dc5-6b3d-4090-9a6c-e7944868a861-kube-api-access-lvmmd\") pod \"test-curl-8024-cffd97\" (UID: \"5bba9dc5-6b3d-4090-9a6c-e7944868a861\") " pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:58:55.725697 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.725669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:55.725885 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.725830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:55.726746 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.726723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:58:55.726840 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.726789    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:58:55.761651 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.761615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmmd\" (UniqueName: \"kubernetes.io/projected/5bba9dc5-6b3d-4090-9a6c-e7944868a861-kube-api-access-lvmmd\") pod \"test-curl-8024-cffd97\" (UID: \"5bba9dc5-6b3d-4090-9a6c-e7944868a861\") " pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:58:55.775581 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.775554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmmd\" (UniqueName: \"kubernetes.io/projected/5bba9dc5-6b3d-4090-9a6c-e7944868a861-kube-api-access-lvmmd\") pod \"test-curl-8024-cffd97\" (UID: \"5bba9dc5-6b3d-4090-9a6c-e7944868a861\") " pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:58:55.909400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:55.909308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:58:56.177142 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:56.177003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8024-cffd97"]
Sep 10 02:58:56.179590 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:58:56.179564    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bba9dc5_6b3d_4090_9a6c_e7944868a861.slice/crio-114e9a6227c7a874fc37f0d82d110a84871342562f44dc8c977e5dcdfd9bd784 WatchSource:0}: Error finding container 114e9a6227c7a874fc37f0d82d110a84871342562f44dc8c977e5dcdfd9bd784: Status 404 returned error can't find the container with id 114e9a6227c7a874fc37f0d82d110a84871342562f44dc8c977e5dcdfd9bd784
Sep 10 02:58:56.730884 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:56.730852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8024-cffd97" event={"ID":"5bba9dc5-6b3d-4090-9a6c-e7944868a861","Type":"ContainerStarted","Data":"114e9a6227c7a874fc37f0d82d110a84871342562f44dc8c977e5dcdfd9bd784"}
Sep 10 02:58:57.518282 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:57.518248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:58:59.747801 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:59.747768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8024-cffd97" event={"ID":"5bba9dc5-6b3d-4090-9a6c-e7944868a861","Type":"ContainerStarted","Data":"3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2"}
Sep 10 02:58:59.762394 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:58:59.762353    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8024-cffd97" podStartSLOduration=1.819973332 podStartE2EDuration="4.762339464s" totalImagesPullingTime="2.942366132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:59.76215613 +0000 UTC m=+736.912055945" watchObservedRunningTime="2026-09-10 02:58:59.762339464 +0000 UTC m=+736.912239160"
Sep 10 02:59:00.388342 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.388308    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8024-cffd97"]
Sep 10 02:59:00.392477 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.392453    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8024-cffd97"]
Sep 10 02:59:00.511744 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.511672    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8024-c091e2"]
Sep 10 02:59:00.512045 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.512033    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" containerName="test-curl-8024-cffd97"
Sep 10 02:59:00.512088 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.512047    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" containerName="test-curl-8024-cffd97"
Sep 10 02:59:00.512123 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.512112    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" containerName="test-curl-8024-cffd97"
Sep 10 02:59:00.515266 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.515250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:00.520843 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.520823    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8024-c091e2"]
Sep 10 02:59:00.610403 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.610373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlwg\" (UniqueName: \"kubernetes.io/projected/a6811d87-6dea-4002-8a88-6b24ed82497b-kube-api-access-jmlwg\") pod \"test-curl-8024-c091e2\" (UID: \"a6811d87-6dea-4002-8a88-6b24ed82497b\") " pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:00.711508 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.711476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlwg\" (UniqueName: \"kubernetes.io/projected/a6811d87-6dea-4002-8a88-6b24ed82497b-kube-api-access-jmlwg\") pod \"test-curl-8024-c091e2\" (UID: \"a6811d87-6dea-4002-8a88-6b24ed82497b\") " pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:00.725993 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.725962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlwg\" (UniqueName: \"kubernetes.io/projected/a6811d87-6dea-4002-8a88-6b24ed82497b-kube-api-access-jmlwg\") pod \"test-curl-8024-c091e2\" (UID: \"a6811d87-6dea-4002-8a88-6b24ed82497b\") " pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:00.752945 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:00.752903    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8024-cffd97" container="test-curl-8024-cffd97"
Sep 10 02:59:00.826778 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:00.826711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:01.094902 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:01.094873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8024-c091e2"]
Sep 10 02:59:01.096606 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:59:01.096583    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6811d87_6dea_4002_8a88_6b24ed82497b.slice/crio-8257f8505a97b15c0cd88fe03d3d8ea762af8b804e98a13407928b08f12d1e59 WatchSource:0}: Error finding container 8257f8505a97b15c0cd88fe03d3d8ea762af8b804e98a13407928b08f12d1e59: Status 404 returned error can't find the container with id 8257f8505a97b15c0cd88fe03d3d8ea762af8b804e98a13407928b08f12d1e59
Sep 10 02:59:01.758521 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:01.758487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8024-c091e2" event={"ID":"a6811d87-6dea-4002-8a88-6b24ed82497b","Type":"ContainerStarted","Data":"b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b"}
Sep 10 02:59:01.758928 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:01.758541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8024-c091e2" event={"ID":"a6811d87-6dea-4002-8a88-6b24ed82497b","Type":"ContainerStarted","Data":"8257f8505a97b15c0cd88fe03d3d8ea762af8b804e98a13407928b08f12d1e59"}
Sep 10 02:59:01.758928 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:01.758549    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8024-cffd97" podUID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" containerName="test-curl-8024-cffd97" containerID="cri-o://3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2" gracePeriod=2
Sep 10 02:59:01.776694 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:01.776650    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8024-c091e2" podStartSLOduration=1.776636406 podStartE2EDuration="1.776636406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:01.775292601 +0000 UTC m=+738.925192299" watchObservedRunningTime="2026-09-10 02:59:01.776636406 +0000 UTC m=+738.926536102"
Sep 10 02:59:02.428372 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.428326    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8024-c091e2"]
Sep 10 02:59:02.435797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.435770    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8024-c091e2"]
Sep 10 02:59:02.607216 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.607157    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"]
Sep 10 02:59:02.607660 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.607643    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6811d87-6dea-4002-8a88-6b24ed82497b" containerName="test-curl-8024-c091e2"
Sep 10 02:59:02.607763 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.607663    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6811d87-6dea-4002-8a88-6b24ed82497b" containerName="test-curl-8024-c091e2"
Sep 10 02:59:02.607822 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.607765    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6811d87-6dea-4002-8a88-6b24ed82497b" containerName="test-curl-8024-c091e2"
Sep 10 02:59:02.610989 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.610968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.613960 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.613938    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-644ff039-openshift-default-dockercfg-xjpdr\""
Sep 10 02:59:02.624207 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.624184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"]
Sep 10 02:59:02.728841 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.728806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-credential-socket\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729009 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.728865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec718091-bd6e-455d-810c-94bad689756c-istiod-ca-cert\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729009 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.728976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-socket\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729095 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.729075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-certs\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729127 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.729115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec718091-bd6e-455d-810c-94bad689756c-istio-podinfo\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729172 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.729141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-data\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729172 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.729158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw96h\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-kube-api-access-rw96h\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729240 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.729224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-envoy\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.729281 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.729241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-istio-token\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.763560 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:02.763532    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8024-c091e2" container="test-curl-8024-c091e2"
Sep 10 02:59:02.830376 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-certs\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830496 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec718091-bd6e-455d-810c-94bad689756c-istio-podinfo\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830496 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-data\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830496 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw96h\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-kube-api-access-rw96h\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830496 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-envoy\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830704 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-istio-token\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830704 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-credential-socket\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830704 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec718091-bd6e-455d-810c-94bad689756c-istiod-ca-cert\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830704 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-socket\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830873 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-certs\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.830920 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-credential-socket\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.831002 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.830984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-socket\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.831133 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.831106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-data\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.831475 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.831456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec718091-bd6e-455d-810c-94bad689756c-istiod-ca-cert\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.833286 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.833262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-envoy\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.833418 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.833401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec718091-bd6e-455d-810c-94bad689756c-istio-podinfo\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.841701 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.841679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw96h\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-kube-api-access-rw96h\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.841776 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.841685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-istio-token\") pod \"e2e-ait-644ff039-openshift-default-649997fd76-gdvfv\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") " pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:02.921714 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:02.921659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:03.198246 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.198218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"]
Sep 10 02:59:03.200655 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:59:03.200619    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec718091_bd6e_455d_810c_94bad689756c.slice/crio-5a2f018fdf3d481d161839ad0d6a4a9ae85716c7afd7bd0af7270c2091b1cf64 WatchSource:0}: Error finding container 5a2f018fdf3d481d161839ad0d6a4a9ae85716c7afd7bd0af7270c2091b1cf64: Status 404 returned error can't find the container with id 5a2f018fdf3d481d161839ad0d6a4a9ae85716c7afd7bd0af7270c2091b1cf64
Sep 10 02:59:03.202973 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.202941    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:03.203088 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.203021    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:03.203088 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.203050    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:03.270995 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.270970    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"]
Sep 10 02:59:03.275213 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.275191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.277900 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.277879    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5ab28b-openshift-default-dockercfg-gxwdx\""
Sep 10 02:59:03.289455 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.289433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"]
Sep 10 02:59:03.335117 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-certs\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335313 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-token\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335313 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-podinfo\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335313 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04541d5c-1857-4df4-8070-00f6e6e34fcf-istiod-ca-cert\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335313 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw59t\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-kube-api-access-mw59t\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335519 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-credential-socket\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335519 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-socket\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335519 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-data\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.335672 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.335521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-envoy\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436205 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-podinfo\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436413 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04541d5c-1857-4df4-8070-00f6e6e34fcf-istiod-ca-cert\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436413 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw59t\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-kube-api-access-mw59t\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436413 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-credential-socket\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436413 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-socket\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436413 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-data\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436673 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-envoy\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436673 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-certs\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436673 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-token\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436840 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-credential-socket\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.436997 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.436973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-certs\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.437432 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.437137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-data\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.437432 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.437390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-socket\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.437584 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.437470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04541d5c-1857-4df4-8070-00f6e6e34fcf-istiod-ca-cert\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.438924 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.438908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-podinfo\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.438997 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.438965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-envoy\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.445132 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.445112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw59t\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-kube-api-access-mw59t\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.445261 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.445143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-token\") pod \"e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") " pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.588933 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.588855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:03.772058 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.772018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" event={"ID":"ec718091-bd6e-455d-810c-94bad689756c","Type":"ContainerStarted","Data":"94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332"}
Sep 10 02:59:03.772510 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.772086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" event={"ID":"ec718091-bd6e-455d-810c-94bad689756c","Type":"ContainerStarted","Data":"5a2f018fdf3d481d161839ad0d6a4a9ae85716c7afd7bd0af7270c2091b1cf64"}
Sep 10 02:59:03.772510 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.772067    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8024-c091e2" podUID="a6811d87-6dea-4002-8a88-6b24ed82497b" containerName="test-curl-8024-c091e2" containerID="cri-o://b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b" gracePeriod=2
Sep 10 02:59:03.794303 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.794257    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podStartSLOduration=1.79424063 podStartE2EDuration="1.79424063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:03.79344429 +0000 UTC m=+740.943343990" watchObservedRunningTime="2026-09-10 02:59:03.79424063 +0000 UTC m=+740.944140326"
Sep 10 02:59:03.868697 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.868668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"]
Sep 10 02:59:03.870937 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:59:03.870912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04541d5c_1857_4df4_8070_00f6e6e34fcf.slice/crio-cc3983bab0758b6affd99940d7ba7a8e0875cbf16a73628cfdccf48cb4352d2a WatchSource:0}: Error finding container cc3983bab0758b6affd99940d7ba7a8e0875cbf16a73628cfdccf48cb4352d2a: Status 404 returned error can't find the container with id cc3983bab0758b6affd99940d7ba7a8e0875cbf16a73628cfdccf48cb4352d2a
Sep 10 02:59:03.873667 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.873635    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:03.873783 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.873695    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:03.873783 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.873725    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:03.922695 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.922659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:03.947903 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:03.947883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:59:04.042064 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.042035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvmmd\" (UniqueName: \"kubernetes.io/projected/5bba9dc5-6b3d-4090-9a6c-e7944868a861-kube-api-access-lvmmd\") pod \"5bba9dc5-6b3d-4090-9a6c-e7944868a861\" (UID: \"5bba9dc5-6b3d-4090-9a6c-e7944868a861\") "
Sep 10 02:59:04.044202 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.044154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bba9dc5-6b3d-4090-9a6c-e7944868a861-kube-api-access-lvmmd" (OuterVolumeSpecName: "kube-api-access-lvmmd") pod "5bba9dc5-6b3d-4090-9a6c-e7944868a861" (UID: "5bba9dc5-6b3d-4090-9a6c-e7944868a861"). InnerVolumeSpecName "kube-api-access-lvmmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:59:04.143063 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.142988    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmmd\" (UniqueName: \"kubernetes.io/projected/5bba9dc5-6b3d-4090-9a6c-e7944868a861-kube-api-access-lvmmd\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:04.778091 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.778060    2580 generic.go:356] "Generic (PLEG): container finished" podID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" containerID="3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2" exitCode=137
Sep 10 02:59:04.778566 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.778110    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:59:04.778566 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.778152    2580 scope.go:117] "RemoveContainer" containerID="3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2"
Sep 10 02:59:04.779912 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.779884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" event={"ID":"04541d5c-1857-4df4-8070-00f6e6e34fcf","Type":"ContainerStarted","Data":"ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887"}
Sep 10 02:59:04.780044 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.779969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" event={"ID":"04541d5c-1857-4df4-8070-00f6e6e34fcf","Type":"ContainerStarted","Data":"cc3983bab0758b6affd99940d7ba7a8e0875cbf16a73628cfdccf48cb4352d2a"}
Sep 10 02:59:04.787824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.787805    2580 scope.go:117] "RemoveContainer" containerID="3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2"
Sep 10 02:59:04.788081 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:04.788065    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2\": container with ID starting with 3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2 not found: ID does not exist" containerID="3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2"
Sep 10 02:59:04.788124 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.788088    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2"} err="failed to get container status \"3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2\": rpc error: code = NotFound desc = could not find container \"3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2\": container with ID starting with 3b8f40c2bc97fdb2305ce34f81e96726b3ca32788e5323f4615fd456ae4b0eb2 not found: ID does not exist"
Sep 10 02:59:04.800647 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.800579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" podStartSLOduration=1.8005635660000001 podStartE2EDuration="1.800563566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:04.79952328 +0000 UTC m=+741.949422976" watchObservedRunningTime="2026-09-10 02:59:04.800563566 +0000 UTC m=+741.950463265"
Sep 10 02:59:04.801815 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:04.801788    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8024-cffd97\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:59:04.926646 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:04.926615    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:04.928874 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:04.928843    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8024-cffd97\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" pod="opendatahub/test-curl-8024-cffd97"
Sep 10 02:59:05.324615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.324581    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bba9dc5-6b3d-4090-9a6c-e7944868a861" path="/var/lib/kubelet/pods/5bba9dc5-6b3d-4090-9a6c-e7944868a861/volumes"
Sep 10 02:59:05.589286 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.589210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:05.594543 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.594518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:05.786654 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.786617    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:05.786953 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.786670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:05.787659 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.787642    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:05.787777 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.787760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:05.953931 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:05.953910    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:06.059939 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.059910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmlwg\" (UniqueName: \"kubernetes.io/projected/a6811d87-6dea-4002-8a88-6b24ed82497b-kube-api-access-jmlwg\") pod \"a6811d87-6dea-4002-8a88-6b24ed82497b\" (UID: \"a6811d87-6dea-4002-8a88-6b24ed82497b\") "
Sep 10 02:59:06.061839 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.061817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6811d87-6dea-4002-8a88-6b24ed82497b-kube-api-access-jmlwg" (OuterVolumeSpecName: "kube-api-access-jmlwg") pod "a6811d87-6dea-4002-8a88-6b24ed82497b" (UID: "a6811d87-6dea-4002-8a88-6b24ed82497b"). InnerVolumeSpecName "kube-api-access-jmlwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:59:06.161465 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.161405    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmlwg\" (UniqueName: \"kubernetes.io/projected/a6811d87-6dea-4002-8a88-6b24ed82497b-kube-api-access-jmlwg\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:06.792194 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.792148    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6811d87-6dea-4002-8a88-6b24ed82497b" containerID="b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b" exitCode=137
Sep 10 02:59:06.792622 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.792209    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:06.792622 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.792219    2580 scope.go:117] "RemoveContainer" containerID="b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b"
Sep 10 02:59:06.800949 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.800931    2580 scope.go:117] "RemoveContainer" containerID="b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b"
Sep 10 02:59:06.801249 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:06.801224    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b\": container with ID starting with b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b not found: ID does not exist" containerID="b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b"
Sep 10 02:59:06.801358 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:06.801256    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b"} err="failed to get container status \"b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b\": rpc error: code = NotFound desc = could not find container \"b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b\": container with ID starting with b3066d2206eb71485b861071519160ed3115007aa6cf4d4414c30d550c20202b not found: ID does not exist"
Sep 10 02:59:06.802244 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:06.802219    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8024-c091e2\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="a6811d87-6dea-4002-8a88-6b24ed82497b" pod="opendatahub/test-curl-8024-c091e2"
Sep 10 02:59:07.325875 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:07.325841    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6811d87-6dea-4002-8a88-6b24ed82497b" path="/var/lib/kubelet/pods/a6811d87-6dea-4002-8a88-6b24ed82497b/volumes"
Sep 10 02:59:08.924189 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:08.923480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"]
Sep 10 02:59:08.940585 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:08.940558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:08.955815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:08.954089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5ab28b-dockercfg-n8zdl\""
Sep 10 02:59:08.955815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:08.954465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5ab28b\""
Sep 10 02:59:08.967539 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:08.967418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"]
Sep 10 02:59:08.991476 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:08.991440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f923b25d-2ac4-4b65-96f3-350c7c6602bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:08.991632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:08.991517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc8jx\" (UniqueName: \"kubernetes.io/projected/f923b25d-2ac4-4b65-96f3-350c7c6602bc-kube-api-access-lc8jx\") pod \"payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:09.092666 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:09.092627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f923b25d-2ac4-4b65-96f3-350c7c6602bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:09.092832 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:09.092677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc8jx\" (UniqueName: \"kubernetes.io/projected/f923b25d-2ac4-4b65-96f3-350c7c6602bc-kube-api-access-lc8jx\") pod \"payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:09.093603 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:09.093583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f923b25d-2ac4-4b65-96f3-350c7c6602bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:09.101773 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:09.101753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc8jx\" (UniqueName: \"kubernetes.io/projected/f923b25d-2ac4-4b65-96f3-350c7c6602bc-kube-api-access-lc8jx\") pod \"payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:09.279772 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:09.279687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:09.601152 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:09.601126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"]
Sep 10 02:59:09.602798 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:59:09.602769    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf923b25d_2ac4_4b65_96f3_350c7c6602bc.slice/crio-ceb4830cd1584e2da00bb9bb3f06ca0c74c2d975f399e7c01371e78858567fcb WatchSource:0}: Error finding container ceb4830cd1584e2da00bb9bb3f06ca0c74c2d975f399e7c01371e78858567fcb: Status 404 returned error can't find the container with id ceb4830cd1584e2da00bb9bb3f06ca0c74c2d975f399e7c01371e78858567fcb
Sep 10 02:59:09.808846 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:09.808798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p" event={"ID":"f923b25d-2ac4-4b65-96f3-350c7c6602bc","Type":"ContainerStarted","Data":"ceb4830cd1584e2da00bb9bb3f06ca0c74c2d975f399e7c01371e78858567fcb"}
Sep 10 02:59:12.882723 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:12.882678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p" event={"ID":"f923b25d-2ac4-4b65-96f3-350c7c6602bc","Type":"ContainerStarted","Data":"35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5"}
Sep 10 02:59:12.883449 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:12.883408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:12.901292 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:12.901242    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p" podStartSLOduration=2.239332327 podStartE2EDuration="4.901227039s" totalImagesPullingTime="2.661894712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:12.899456708 +0000 UTC m=+750.049356405" watchObservedRunningTime="2026-09-10 02:59:12.901227039 +0000 UTC m=+750.051126727"
Sep 10 02:59:15.812812 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:15.812769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:59:16.908001 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.907963    2580 generic.go:356] "Generic (PLEG): container finished" podID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerID="7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15" exitCode=0
Sep 10 02:59:16.908564 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.908030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" event={"ID":"96ec6504-208f-4bda-a4d8-2ae1a03203a0","Type":"ContainerDied","Data":"7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15"}
Sep 10 02:59:16.908564 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.908518    2580 scope.go:117] "RemoveContainer" containerID="7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15"
Sep 10 02:59:16.909649 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.909599    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:16.909781 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.909687    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:16.909781 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.909733    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:16.909931 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.909909    2580 generic.go:356] "Generic (PLEG): container finished" podID="61779f37-235f-4b0d-b030-8c4c125dac42" containerID="d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37" exitCode=0
Sep 10 02:59:16.910001 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.909979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" event={"ID":"61779f37-235f-4b0d-b030-8c4c125dac42","Type":"ContainerDied","Data":"d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37"}
Sep 10 02:59:16.910345 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.910327    2580 scope.go:117] "RemoveContainer" containerID="d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37"
Sep 10 02:59:16.911297 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.911271    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:16.911403 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.911329    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:16.911403 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:16.911359    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:17.923560 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:17.923517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" event={"ID":"96ec6504-208f-4bda-a4d8-2ae1a03203a0","Type":"ContainerStarted","Data":"20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8"}
Sep 10 02:59:17.927602 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:17.927556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" event={"ID":"61779f37-235f-4b0d-b030-8c4c125dac42","Type":"ContainerStarted","Data":"bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d"}
Sep 10 02:59:18.074228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:18.073797    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:59:18.074500 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:18.074376    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 02:59:18.074500 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:18.074440    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 02:59:18.217909 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:18.217397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:59:18.218110 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:18.217999    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 02:59:18.218110 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:18.218079    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 02:59:19.073258 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:19.073211    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 02:59:19.073744 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:19.073293    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 02:59:19.217638 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:19.217591    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 02:59:19.217843 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:19.217663    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 02:59:20.073073 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:20.073027    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 02:59:20.073306 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:20.073096    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 02:59:20.217924 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:20.217878    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 02:59:20.218138 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:20.217961    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 02:59:21.073364 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:21.073318    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 02:59:21.073867 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:21.073388    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 02:59:21.218311 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:21.218255    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 02:59:21.218587 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:21.218558    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 02:59:22.073514 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:22.073461    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 02:59:22.074123 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:22.074073    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 02:59:22.218455 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:22.218419    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 02:59:22.218767 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:22.218719    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 02:59:23.073020 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:23.072985    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 02:59:23.073352 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:23.073315    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 02:59:23.217817 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:23.217782    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 02:59:23.218469 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:23.218410    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 02:59:24.077524 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:24.077497    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:59:24.077834 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:24.077805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:59:24.078872 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:24.078819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 02:59:24.222624 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:24.222585    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:59:24.223089 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:24.222993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:59:24.223681 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:24.223660    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 02:59:24.896600 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:24.896567    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 02:59:26.997372 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:26.997341    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec718091-bd6e-455d-810c-94bad689756c" containerID="94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332" exitCode=0
Sep 10 02:59:26.998768 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:26.997419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" event={"ID":"ec718091-bd6e-455d-810c-94bad689756c","Type":"ContainerDied","Data":"94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332"}
Sep 10 02:59:26.998768 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:26.997866    2580 scope.go:117] "RemoveContainer" containerID="94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332"
Sep 10 02:59:26.999291 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:26.999017    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:26.999291 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:26.999098    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:26.999291 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:26.999151    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:27.793272 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.793235    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"]
Sep 10 02:59:27.796263 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.796244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.799400 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.799381    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ce5f9d-openshift-default-dockercfg-2bc6d\""
Sep 10 02:59:27.824187 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.824053    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"]
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-socket\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qgzj\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-kube-api-access-5qgzj\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b674bd-3112-4d6c-956b-91c5be7739f8-istiod-ca-cert\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-certs\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-data\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-envoy\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-token\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-credential-socket\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.838334 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.838313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-podinfo\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939093 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b674bd-3112-4d6c-956b-91c5be7739f8-istiod-ca-cert\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939314 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-certs\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939314 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-data\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939314 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-envoy\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939314 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-token\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939314 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-credential-socket\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939314 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-podinfo\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-socket\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qgzj\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-kube-api-access-5qgzj\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939615 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-certs\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.939954 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b674bd-3112-4d6c-956b-91c5be7739f8-istiod-ca-cert\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.940092 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.939980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-data\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.940092 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.940069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-socket\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.940231 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.940208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-credential-socket\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.942041 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.942019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-envoy\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.942704 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.942680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-podinfo\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.947891 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.947868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-token\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:27.948051 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:27.948034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qgzj\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-kube-api-access-5qgzj\") pod \"e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") " pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:28.005710 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.005676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" event={"ID":"ec718091-bd6e-455d-810c-94bad689756c","Type":"ContainerStarted","Data":"c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d"}
Sep 10 02:59:28.007537 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.007516    2580 generic.go:356] "Generic (PLEG): container finished" podID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerID="ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887" exitCode=0
Sep 10 02:59:28.007625 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.007564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" event={"ID":"04541d5c-1857-4df4-8070-00f6e6e34fcf","Type":"ContainerDied","Data":"ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887"}
Sep 10 02:59:28.007883 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.007871    2580 scope.go:117] "RemoveContainer" containerID="ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887"
Sep 10 02:59:28.014064 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.013726    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:28.014064 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.013812    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:28.014064 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.013857    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:28.106808 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.106781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:28.413772 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.413723    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"]
Sep 10 02:59:28.422362 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.422316    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:28.422472 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.422438    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:28.422533 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.422485    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:28.922632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.922588    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:28.923122 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.923093    2580 patch_prober.go:28] interesting pod/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 02:59:28.923450 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:28.923158    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 02:59:29.036187 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.035551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" event={"ID":"65b674bd-3112-4d6c-956b-91c5be7739f8","Type":"ContainerStarted","Data":"93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82"}
Sep 10 02:59:29.036187 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.035624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" event={"ID":"65b674bd-3112-4d6c-956b-91c5be7739f8","Type":"ContainerStarted","Data":"4d7d0a69b2aeb8bbbba4bd32e62e5cd3ee492244d8ce8398d9237648a8f9f8fd"}
Sep 10 02:59:29.041186 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.040706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" event={"ID":"04541d5c-1857-4df4-8070-00f6e6e34fcf","Type":"ContainerStarted","Data":"e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40"}
Sep 10 02:59:29.086660 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.086605    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podStartSLOduration=2.086590402 podStartE2EDuration="2.086590402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:29.065519388 +0000 UTC m=+766.215419085" watchObservedRunningTime="2026-09-10 02:59:29.086590402 +0000 UTC m=+766.236490100"
Sep 10 02:59:29.107123 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.107097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:29.108708 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.108675    2580 patch_prober.go:28] interesting pod/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 02:59:29.108815 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.108734    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 02:59:29.589035 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.588981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:29.589345 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.589291    2580 patch_prober.go:28] interesting pod/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 10 02:59:29.589420 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.589378    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 10 02:59:29.922796 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.922354    2580 patch_prober.go:28] interesting pod/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 02:59:29.922796 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:29.922418    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 02:59:30.124048 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.124004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:30.589506 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.589472    2580 patch_prober.go:28] interesting pod/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 10 02:59:30.589685 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.589529    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 10 02:59:30.700660 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.700623    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"]
Sep 10 02:59:30.703076 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.703060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"
Sep 10 02:59:30.705877 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.705706    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 02:59:30.705877 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.705822    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 02:59:30.714012 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.713973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"]
Sep 10 02:59:30.768808 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.768741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041516d4-c68e-40c4-a255-16bfa17adeb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-644ff039-jgtnj\" (UID: \"041516d4-c68e-40c4-a255-16bfa17adeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"
Sep 10 02:59:30.870260 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.870216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041516d4-c68e-40c4-a255-16bfa17adeb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-644ff039-jgtnj\" (UID: \"041516d4-c68e-40c4-a255-16bfa17adeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"
Sep 10 02:59:30.870829 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.870808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041516d4-c68e-40c4-a255-16bfa17adeb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-644ff039-jgtnj\" (UID: \"041516d4-c68e-40c4-a255-16bfa17adeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"
Sep 10 02:59:30.922973 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.922934    2580 patch_prober.go:28] interesting pod/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 02:59:30.923296 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:30.923266    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 02:59:31.015699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.015652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"
Sep 10 02:59:31.061483 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.061458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:31.066441 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.066418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:31.337643 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.337558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"]
Sep 10 02:59:31.589557 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.589473    2580 patch_prober.go:28] interesting pod/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 10 02:59:31.589557 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.589533    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 10 02:59:31.923365 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.922892    2580 patch_prober.go:28] interesting pod/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 02:59:31.923365 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:31.922957    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 02:59:32.074012 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.073960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj" event={"ID":"041516d4-c68e-40c4-a255-16bfa17adeb0","Type":"ContainerStarted","Data":"14a1a109d5a03451f420d8c4aad8a14acbf044e706db14d848220e2354039a80"}
Sep 10 02:59:32.590271 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.590235    2580 patch_prober.go:28] interesting pod/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 10 02:59:32.590864 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.590304    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 10 02:59:32.624980 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.624940    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:59:32.831657 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.830860    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"]
Sep 10 02:59:32.848065 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.847897    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"]
Sep 10 02:59:32.848366 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.848030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:32.855765 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.854440    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ce5f9d\""
Sep 10 02:59:32.878790 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.878762    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"]
Sep 10 02:59:32.881402 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.881381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:32.884686 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.884632    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ce5f9d-dockercfg-gcp7m\""
Sep 10 02:59:32.893123 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.893095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"]
Sep 10 02:59:32.900838 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.900775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec566af6-700d-4672-a732-c7b0950db52d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:32.900970 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.900871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvj4\" (UniqueName: \"kubernetes.io/projected/ec566af6-700d-4672-a732-c7b0950db52d-kube-api-access-wvvj4\") pod \"payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:32.928189 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.926625    2580 patch_prober.go:28] interesting pod/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 02:59:32.928189 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:32.926709    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 02:59:33.002410 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.002352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c23839-197c-4004-a0ed-2541c16f2a5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:33.002410 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.002401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gjc\" (UniqueName: \"kubernetes.io/projected/49c23839-197c-4004-a0ed-2541c16f2a5c-kube-api-access-v7gjc\") pod \"payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:33.002632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.002508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec566af6-700d-4672-a732-c7b0950db52d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:33.002632 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.002542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvj4\" (UniqueName: \"kubernetes.io/projected/ec566af6-700d-4672-a732-c7b0950db52d-kube-api-access-wvvj4\") pod \"payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:33.003299 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.003278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec566af6-700d-4672-a732-c7b0950db52d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:33.049189 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.048407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvj4\" (UniqueName: \"kubernetes.io/projected/ec566af6-700d-4672-a732-c7b0950db52d-kube-api-access-wvvj4\") pod \"payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:33.103506 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.103414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c23839-197c-4004-a0ed-2541c16f2a5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:33.103506 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.103458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gjc\" (UniqueName: \"kubernetes.io/projected/49c23839-197c-4004-a0ed-2541c16f2a5c-kube-api-access-v7gjc\") pod \"payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:33.104366 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.104341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c23839-197c-4004-a0ed-2541c16f2a5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:33.131187 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.130821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gjc\" (UniqueName: \"kubernetes.io/projected/49c23839-197c-4004-a0ed-2541c16f2a5c-kube-api-access-v7gjc\") pod \"payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:33.175637 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.175589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:33.193589 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.193376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:33.500464 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.500182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"]
Sep 10 02:59:33.503973 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:59:33.503942    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec566af6_700d_4672_a732_c7b0950db52d.slice/crio-80758eb630b4c31e2ae5aa1209c672ccf4a829455767ab2a53c38d58c1d85599 WatchSource:0}: Error finding container 80758eb630b4c31e2ae5aa1209c672ccf4a829455767ab2a53c38d58c1d85599: Status 404 returned error can't find the container with id 80758eb630b4c31e2ae5aa1209c672ccf4a829455767ab2a53c38d58c1d85599
Sep 10 02:59:33.519839 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.519803    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"]
Sep 10 02:59:33.542835 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:59:33.542787    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c23839_197c_4004_a0ed_2541c16f2a5c.slice/crio-03c3ee572ec1fb8fe7c7caf21d82dc8d5c97db5093127437eba4112b18b30665 WatchSource:0}: Error finding container 03c3ee572ec1fb8fe7c7caf21d82dc8d5c97db5093127437eba4112b18b30665: Status 404 returned error can't find the container with id 03c3ee572ec1fb8fe7c7caf21d82dc8d5c97db5093127437eba4112b18b30665
Sep 10 02:59:33.591042 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.591015    2580 patch_prober.go:28] interesting pod/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 10 02:59:33.592047 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.591644    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 10 02:59:33.941422 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.941387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:33.942374 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.942311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:33.943180 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:33.943138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:34.087610 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.087572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb" event={"ID":"49c23839-197c-4004-a0ed-2541c16f2a5c","Type":"ContainerStarted","Data":"faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5"}
Sep 10 02:59:34.087771 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.087639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb" event={"ID":"49c23839-197c-4004-a0ed-2541c16f2a5c","Type":"ContainerStarted","Data":"03c3ee572ec1fb8fe7c7caf21d82dc8d5c97db5093127437eba4112b18b30665"}
Sep 10 02:59:34.087771 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.087703    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:34.089524 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.089498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh" event={"ID":"ec566af6-700d-4672-a732-c7b0950db52d","Type":"ContainerStarted","Data":"b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426"}
Sep 10 02:59:34.089653 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.089545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh" event={"ID":"ec566af6-700d-4672-a732-c7b0950db52d","Type":"ContainerStarted","Data":"80758eb630b4c31e2ae5aa1209c672ccf4a829455767ab2a53c38d58c1d85599"}
Sep 10 02:59:34.089717 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.089675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:34.108137 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.108079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb" podStartSLOduration=2.108061023 podStartE2EDuration="2.108061023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:34.10477882 +0000 UTC m=+771.254678528" watchObservedRunningTime="2026-09-10 02:59:34.108061023 +0000 UTC m=+771.257960721"
Sep 10 02:59:34.129560 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.129484    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh" podStartSLOduration=2.129463426 podStartE2EDuration="2.129463426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:34.126028534 +0000 UTC m=+771.275928256" watchObservedRunningTime="2026-09-10 02:59:34.129463426 +0000 UTC m=+771.279363125"
Sep 10 02:59:34.593689 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.593659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:34.594110 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.593957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:34.594942 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:34.594921    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 02:59:35.095569 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:35.095532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj" event={"ID":"041516d4-c68e-40c4-a255-16bfa17adeb0","Type":"ContainerStarted","Data":"6415172f0ee884f35e60f07a1607ede19ef1037bcc67cd526a92767f185b6fe4"}
Sep 10 02:59:35.113283 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:35.113233    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj" podStartSLOduration=1.8224183699999998 podStartE2EDuration="5.113213363s" totalImagesPullingTime="3.290794993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:35.11154567 +0000 UTC m=+772.261445374" watchObservedRunningTime="2026-09-10 02:59:35.113213363 +0000 UTC m=+772.263113061"
Sep 10 02:59:36.110108 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.110070    2580 generic.go:356] "Generic (PLEG): container finished" podID="041516d4-c68e-40c4-a255-16bfa17adeb0" containerID="6415172f0ee884f35e60f07a1607ede19ef1037bcc67cd526a92767f185b6fe4" exitCode=0
Sep 10 02:59:36.110589 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.110152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj" event={"ID":"041516d4-c68e-40c4-a255-16bfa17adeb0","Type":"ContainerDied","Data":"6415172f0ee884f35e60f07a1607ede19ef1037bcc67cd526a92767f185b6fe4"}
Sep 10 02:59:36.814084 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.814052    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"]
Sep 10 02:59:36.816562 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.816544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:36.824857 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.824832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"kube-root-ca.crt\""
Sep 10 02:59:36.824980 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.824871    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"default-dockercfg-lj6l4\""
Sep 10 02:59:36.824980 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.824890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"openshift-service-ca.crt\""
Sep 10 02:59:36.824980 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.824910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"simulator-w0-097e16-kserve-self-signed-certs\""
Sep 10 02:59:36.836876 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.836851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"]
Sep 10 02:59:36.959520 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.959478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7qq\" (UniqueName: \"kubernetes.io/projected/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-kube-api-access-2f7qq\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:36.959667 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.959613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tmp-dir\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:36.959667 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.959640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-home\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:36.959774 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.959677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-dshm\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:36.959813 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.959766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-model-cache\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:36.959813 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:36.959796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tls-certs\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060293 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7qq\" (UniqueName: \"kubernetes.io/projected/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-kube-api-access-2f7qq\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060448 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tmp-dir\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060448 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-home\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060448 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-dshm\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060448 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-model-cache\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060448 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tls-certs\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060775 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tmp-dir\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060832 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-home\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.060899 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.060880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-model-cache\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.062607 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.062584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-dshm\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.062881 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.062864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tls-certs\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.069480 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.069423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7qq\" (UniqueName: \"kubernetes.io/projected/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-kube-api-access-2f7qq\") pod \"simulator-w0-097e16-kserve-8bf9dc58f-fzs2w\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") " pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.125305 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.125277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:37.257414 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.257393    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"
Sep 10 02:59:37.363059 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.363034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041516d4-c68e-40c4-a255-16bfa17adeb0-maas-api-service-ca\") pod \"041516d4-c68e-40c4-a255-16bfa17adeb0\" (UID: \"041516d4-c68e-40c4-a255-16bfa17adeb0\") "
Sep 10 02:59:37.363390 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.363367    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041516d4-c68e-40c4-a255-16bfa17adeb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "041516d4-c68e-40c4-a255-16bfa17adeb0" (UID: "041516d4-c68e-40c4-a255-16bfa17adeb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:59:37.418077 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.418055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"]
Sep 10 02:59:37.419703 ip-10-0-142-156 kubenswrapper[2580]: W0910 02:59:37.419674    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc48b3c6_fb53_4d4f_80ac_166b8c3b4cab.slice/crio-294c5791c6e2619fc0558d07a8e4d1ff843be46f076eadb2aaace8a0c6c4fd0a WatchSource:0}: Error finding container 294c5791c6e2619fc0558d07a8e4d1ff843be46f076eadb2aaace8a0c6c4fd0a: Status 404 returned error can't find the container with id 294c5791c6e2619fc0558d07a8e4d1ff843be46f076eadb2aaace8a0c6c4fd0a
Sep 10 02:59:37.463919 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:37.463891    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041516d4-c68e-40c4-a255-16bfa17adeb0-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:38.122947 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:38.122911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w" event={"ID":"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab","Type":"ContainerStarted","Data":"688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838"}
Sep 10 02:59:38.123101 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:38.122957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w" event={"ID":"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab","Type":"ContainerStarted","Data":"294c5791c6e2619fc0558d07a8e4d1ff843be46f076eadb2aaace8a0c6c4fd0a"}
Sep 10 02:59:38.124201 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:38.124154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj" event={"ID":"041516d4-c68e-40c4-a255-16bfa17adeb0","Type":"ContainerDied","Data":"14a1a109d5a03451f420d8c4aad8a14acbf044e706db14d848220e2354039a80"}
Sep 10 02:59:38.124294 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:38.124223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"
Sep 10 02:59:38.124340 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:38.124224    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a1a109d5a03451f420d8c4aad8a14acbf044e706db14d848220e2354039a80"
Sep 10 02:59:38.145635 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:38.145594    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w" podStartSLOduration=2.145580452 podStartE2EDuration="2.145580452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:38.143192322 +0000 UTC m=+775.293092012" watchObservedRunningTime="2026-09-10 02:59:38.145580452 +0000 UTC m=+775.295480148"
Sep 10 02:59:45.098021 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:45.097981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 02:59:45.098395 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:45.098070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 02:59:47.126602 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.125764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:47.126602 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.125839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:47.143699 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.143674    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:47.193379 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.193355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 02:59:47.511285 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.511196    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"]
Sep 10 02:59:47.514516 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.514485    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-644ff039-jgtnj"]
Sep 10 02:59:47.655986 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.655955    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"]
Sep 10 02:59:47.657228 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:47.656990    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy" containerID="cri-o://c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d" gracePeriod=30
Sep 10 02:59:48.944856 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:48.944811    2580 patch_prober.go:28] interesting pod/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 02:59:48.945267 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:48.944890    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 02:59:49.328209 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:49.328111    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041516d4-c68e-40c4-a255-16bfa17adeb0" path="/var/lib/kubelet/pods/041516d4-c68e-40c4-a255-16bfa17adeb0/volumes"
Sep 10 02:59:51.209224 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:51.209104    2580 generic.go:356] "Generic (PLEG): container finished" podID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerID="93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82" exitCode=0
Sep 10 02:59:51.209224 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:51.209201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" event={"ID":"65b674bd-3112-4d6c-956b-91c5be7739f8","Type":"ContainerDied","Data":"93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82"}
Sep 10 02:59:51.209750 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:51.209593    2580 scope.go:117] "RemoveContainer" containerID="93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82"
Sep 10 02:59:51.210741 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:51.210714    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:51.210807 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:51.210778    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:51.210843 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:51.210814    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:59:52.216374 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:52.216340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" event={"ID":"65b674bd-3112-4d6c-956b-91c5be7739f8","Type":"ContainerStarted","Data":"b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba"}
Sep 10 02:59:52.723452 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:52.723408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 02:59:52.943824 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:52.943752    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:53.107517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.107483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:53.107797 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.107777    2580 patch_prober.go:28] interesting pod/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 02:59:53.107855 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.107821    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 02:59:53.136355 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-credential-socket\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw96h\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-kube-api-access-rw96h\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-data\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136517 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-envoy\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136665 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec718091-bd6e-455d-810c-94bad689756c-istio-podinfo\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136665 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136575    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec718091-bd6e-455d-810c-94bad689756c-istiod-ca-cert\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136665 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-certs\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136665 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:59:53.136809 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-istio-token\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136809 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-socket\") pod \"ec718091-bd6e-455d-810c-94bad689756c\" (UID: \"ec718091-bd6e-455d-810c-94bad689756c\") "
Sep 10 02:59:53.136809 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:59:53.137042 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec718091-bd6e-455d-810c-94bad689756c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:59:53.137042 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.136881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:59:53.137042 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.137014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:59:53.137220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.137117    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.137220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.137130    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.137220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.137144    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.137220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.137177    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec718091-bd6e-455d-810c-94bad689756c-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.137220 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.137194    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.138909 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.138878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-kube-api-access-rw96h" (OuterVolumeSpecName: "kube-api-access-rw96h") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "kube-api-access-rw96h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:59:53.139023 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.138950    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:59:53.139023 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.138947    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec718091-bd6e-455d-810c-94bad689756c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 02:59:53.139023 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.138994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec718091-bd6e-455d-810c-94bad689756c" (UID: "ec718091-bd6e-455d-810c-94bad689756c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:59:53.227952 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.227845    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec718091-bd6e-455d-810c-94bad689756c" containerID="c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d" exitCode=0
Sep 10 02:59:53.228449 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.227999    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"
Sep 10 02:59:53.231248 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.231204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" event={"ID":"ec718091-bd6e-455d-810c-94bad689756c","Type":"ContainerDied","Data":"c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d"}
Sep 10 02:59:53.234471 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.234434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv" event={"ID":"ec718091-bd6e-455d-810c-94bad689756c","Type":"ContainerDied","Data":"5a2f018fdf3d481d161839ad0d6a4a9ae85716c7afd7bd0af7270c2091b1cf64"}
Sep 10 02:59:53.234612 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.234496    2580 scope.go:117] "RemoveContainer" containerID="c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d"
Sep 10 02:59:53.243185 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.238271    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec718091-bd6e-455d-810c-94bad689756c-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.243185 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.238307    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.243185 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.238322    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw96h\" (UniqueName: \"kubernetes.io/projected/ec718091-bd6e-455d-810c-94bad689756c-kube-api-access-rw96h\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.243185 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.238336    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec718091-bd6e-455d-810c-94bad689756c-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 02:59:53.248737 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.248713    2580 scope.go:117] "RemoveContainer" containerID="94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332"
Sep 10 02:59:53.257040 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.257023    2580 scope.go:117] "RemoveContainer" containerID="c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d"
Sep 10 02:59:53.257382 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:53.257365    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d\": container with ID starting with c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d not found: ID does not exist" containerID="c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d"
Sep 10 02:59:53.257433 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.257389    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d"} err="failed to get container status \"c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d\": rpc error: code = NotFound desc = could not find container \"c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d\": container with ID starting with c9468e12df43bb0b7cdf05839956591bf434bae618c5b38b010ac74257916b6d not found: ID does not exist"
Sep 10 02:59:53.257433 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.257405    2580 scope.go:117] "RemoveContainer" containerID="94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332"
Sep 10 02:59:53.257629 ip-10-0-142-156 kubenswrapper[2580]: E0910 02:59:53.257612    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332\": container with ID starting with 94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332 not found: ID does not exist" containerID="94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332"
Sep 10 02:59:53.257673 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.257632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332"} err="failed to get container status \"94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332\": rpc error: code = NotFound desc = could not find container \"94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332\": container with ID starting with 94ac6745b0487abea20962014c0280019096cc7cf41384b0b78a7ddfb15d0332 not found: ID does not exist"
Sep 10 02:59:53.263991 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.263972    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"]
Sep 10 02:59:53.270121 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.270098    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-644ff039-openshift-default-649997fd76-gdvfv"]
Sep 10 02:59:53.334235 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:53.334187    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec718091-bd6e-455d-810c-94bad689756c" path="/var/lib/kubelet/pods/ec718091-bd6e-455d-810c-94bad689756c/volumes"
Sep 10 02:59:54.108009 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:54.107972    2580 patch_prober.go:28] interesting pod/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 02:59:54.108280 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:54.108030    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 02:59:55.107751 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:55.107712    2580 patch_prober.go:28] interesting pod/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 02:59:55.108252 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:55.107770    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 02:59:56.107480 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:56.107445    2580 patch_prober.go:28] interesting pod/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 02:59:56.107636 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:56.107499    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 02:59:57.107481 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:57.107444    2580 patch_prober.go:28] interesting pod/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 02:59:57.107879 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:57.107503    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 02:59:58.182848 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:58.182817    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:58.183339 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:58.183270    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 02:59:58.184407 ip-10-0-142-156 kubenswrapper[2580]: I0910 02:59:58.184387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 03:00:00.136882 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.136845    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"]
Sep 10 03:00:00.137392 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137374    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy"
Sep 10 03:00:00.137485 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137395    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy"
Sep 10 03:00:00.137485 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137411    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy"
Sep 10 03:00:00.137485 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137418    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy"
Sep 10 03:00:00.137485 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137441    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041516d4-c68e-40c4-a255-16bfa17adeb0" containerName="revoke-keys"
Sep 10 03:00:00.137485 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137449    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="041516d4-c68e-40c4-a255-16bfa17adeb0" containerName="revoke-keys"
Sep 10 03:00:00.137730 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137568    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="041516d4-c68e-40c4-a255-16bfa17adeb0" containerName="revoke-keys"
Sep 10 03:00:00.137730 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137584    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy"
Sep 10 03:00:00.137730 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.137595    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec718091-bd6e-455d-810c-94bad689756c" containerName="istio-proxy"
Sep 10 03:00:00.145264 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.145244    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"]
Sep 10 03:00:00.145373 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.145362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"
Sep 10 03:00:00.148773 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.148750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:00:00.150959 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.149532    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq"]
Sep 10 03:00:00.150959 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.149767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"
Sep 10 03:00:00.156413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.154738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"]
Sep 10 03:00:00.156413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.154775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"]
Sep 10 03:00:00.156413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.154853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq"
Sep 10 03:00:00.156413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.156244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq"]
Sep 10 03:00:00.163650 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.163628    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"]
Sep 10 03:00:00.163767 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.163707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"
Sep 10 03:00:00.167409 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.167388    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"
Sep 10 03:00:00.173784 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.173761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"
Sep 10 03:00:00.175910 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.175877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"]
Sep 10 03:00:00.221941 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.221300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq"
Sep 10 03:00:00.235098 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.234886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"
Sep 10 03:00:00.446052 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.446021    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"]
Sep 10 03:00:00.446984 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:00.446964    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029d658e_d8c2_47f4_87b8_1fb69731cd43.slice/crio-ef5371c09110ccb632b982a878e8acf0b1303056f52c2de1ff7bd1ec54e6d96b WatchSource:0}: Error finding container ef5371c09110ccb632b982a878e8acf0b1303056f52c2de1ff7bd1ec54e6d96b: Status 404 returned error can't find the container with id ef5371c09110ccb632b982a878e8acf0b1303056f52c2de1ff7bd1ec54e6d96b
Sep 10 03:00:00.461607 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.461579    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"]
Sep 10 03:00:00.462541 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:00.462519    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf32dce_3382_48c2_baec_fca97d1a866e.slice/crio-98359651b7da8b7df74d1910846a5d3f57e170b0ef03f28c90ad8c309604b9f7 WatchSource:0}: Error finding container 98359651b7da8b7df74d1910846a5d3f57e170b0ef03f28c90ad8c309604b9f7: Status 404 returned error can't find the container with id 98359651b7da8b7df74d1910846a5d3f57e170b0ef03f28c90ad8c309604b9f7
Sep 10 03:00:00.527716 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.527696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq"]
Sep 10 03:00:00.529184 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:00.529144    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2464ac4_e3a4_48d7_bdea_7f64e1fc5bef.slice/crio-0a9badeac6753cb649c01d392e8a467e3090c3f96f706587bb35a1e5f0522b44 WatchSource:0}: Error finding container 0a9badeac6753cb649c01d392e8a467e3090c3f96f706587bb35a1e5f0522b44: Status 404 returned error can't find the container with id 0a9badeac6753cb649c01d392e8a467e3090c3f96f706587bb35a1e5f0522b44
Sep 10 03:00:00.536687 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:00.536657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"]
Sep 10 03:00:00.538468 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:00.538446    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468ec338_2565_4c37_8616_8a75622b08d7.slice/crio-4be6aafc03d60cda2c6cfeb01cf49acb7f22f810b0cda2f5b8184af0d2e6a921 WatchSource:0}: Error finding container 4be6aafc03d60cda2c6cfeb01cf49acb7f22f810b0cda2f5b8184af0d2e6a921: Status 404 returned error can't find the container with id 4be6aafc03d60cda2c6cfeb01cf49acb7f22f810b0cda2f5b8184af0d2e6a921
Sep 10 03:00:01.279619 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.279582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq" event={"ID":"a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef","Type":"ContainerStarted","Data":"440662120e70e238384d5422486bf828f1a1d2823da8e38a97ae2ce4bf8510a4"}
Sep 10 03:00:01.280077 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.279632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq" event={"ID":"a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef","Type":"ContainerStarted","Data":"0a9badeac6753cb649c01d392e8a467e3090c3f96f706587bb35a1e5f0522b44"}
Sep 10 03:00:01.280956 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.280934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p" event={"ID":"adf32dce-3382-48c2-baec-fca97d1a866e","Type":"ContainerStarted","Data":"b972a8442d8acfacc215847683f8c3767cc96b5eef2fe2dc3671a9dff63cab5a"}
Sep 10 03:00:01.281044 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.280984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p" event={"ID":"adf32dce-3382-48c2-baec-fca97d1a866e","Type":"ContainerStarted","Data":"98359651b7da8b7df74d1910846a5d3f57e170b0ef03f28c90ad8c309604b9f7"}
Sep 10 03:00:01.282410 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.282387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz" event={"ID":"029d658e-d8c2-47f4-87b8-1fb69731cd43","Type":"ContainerStarted","Data":"bd560ee19b30d1836cbc28261e47bcd46bdddfdcc718b43fcbceca906bf89d92"}
Sep 10 03:00:01.282516 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.282424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz" event={"ID":"029d658e-d8c2-47f4-87b8-1fb69731cd43","Type":"ContainerStarted","Data":"ef5371c09110ccb632b982a878e8acf0b1303056f52c2de1ff7bd1ec54e6d96b"}
Sep 10 03:00:01.283792 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.283759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8" event={"ID":"468ec338-2565-4c37-8616-8a75622b08d7","Type":"ContainerStarted","Data":"b3cdc2556d3692e16e00ac5fc11739472ad249dac859dfc4fa5cf5ba49920867"}
Sep 10 03:00:01.283863 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.283804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8" event={"ID":"468ec338-2565-4c37-8616-8a75622b08d7","Type":"ContainerStarted","Data":"4be6aafc03d60cda2c6cfeb01cf49acb7f22f810b0cda2f5b8184af0d2e6a921"}
Sep 10 03:00:01.294804 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.294765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq" podStartSLOduration=1.294752727 podStartE2EDuration="1.294752727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:01.294339727 +0000 UTC m=+798.444239424" watchObservedRunningTime="2026-09-10 03:00:01.294752727 +0000 UTC m=+798.444652448"
Sep 10 03:00:01.310699 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.310658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz" podStartSLOduration=1.310645569 podStartE2EDuration="1.310645569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:01.309484667 +0000 UTC m=+798.459384364" watchObservedRunningTime="2026-09-10 03:00:01.310645569 +0000 UTC m=+798.460545266"
Sep 10 03:00:01.331995 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.331789    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8" podStartSLOduration=1.331774196 podStartE2EDuration="1.331774196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:01.329037256 +0000 UTC m=+798.478936954" watchObservedRunningTime="2026-09-10 03:00:01.331774196 +0000 UTC m=+798.481673896"
Sep 10 03:00:01.353921 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:01.353872    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p" podStartSLOduration=1.353858625 podStartE2EDuration="1.353858625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:01.352378617 +0000 UTC m=+798.502278314" watchObservedRunningTime="2026-09-10 03:00:01.353858625 +0000 UTC m=+798.503758320"
Sep 10 03:00:02.290917 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.290881    2580 generic.go:356] "Generic (PLEG): container finished" podID="a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef" containerID="440662120e70e238384d5422486bf828f1a1d2823da8e38a97ae2ce4bf8510a4" exitCode=0
Sep 10 03:00:02.291384 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.290962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq" event={"ID":"a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef","Type":"ContainerDied","Data":"440662120e70e238384d5422486bf828f1a1d2823da8e38a97ae2ce4bf8510a4"}
Sep 10 03:00:02.292446 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.292422    2580 generic.go:356] "Generic (PLEG): container finished" podID="adf32dce-3382-48c2-baec-fca97d1a866e" containerID="b972a8442d8acfacc215847683f8c3767cc96b5eef2fe2dc3671a9dff63cab5a" exitCode=0
Sep 10 03:00:02.292542 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.292497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p" event={"ID":"adf32dce-3382-48c2-baec-fca97d1a866e","Type":"ContainerDied","Data":"b972a8442d8acfacc215847683f8c3767cc96b5eef2fe2dc3671a9dff63cab5a"}
Sep 10 03:00:02.293733 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.293712    2580 generic.go:356] "Generic (PLEG): container finished" podID="029d658e-d8c2-47f4-87b8-1fb69731cd43" containerID="bd560ee19b30d1836cbc28261e47bcd46bdddfdcc718b43fcbceca906bf89d92" exitCode=0
Sep 10 03:00:02.293808 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.293788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz" event={"ID":"029d658e-d8c2-47f4-87b8-1fb69731cd43","Type":"ContainerDied","Data":"bd560ee19b30d1836cbc28261e47bcd46bdddfdcc718b43fcbceca906bf89d92"}
Sep 10 03:00:02.295346 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.295323    2580 generic.go:356] "Generic (PLEG): container finished" podID="468ec338-2565-4c37-8616-8a75622b08d7" containerID="b3cdc2556d3692e16e00ac5fc11739472ad249dac859dfc4fa5cf5ba49920867" exitCode=0
Sep 10 03:00:02.295441 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:02.295398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8" event={"ID":"468ec338-2565-4c37-8616-8a75622b08d7","Type":"ContainerDied","Data":"b3cdc2556d3692e16e00ac5fc11739472ad249dac859dfc4fa5cf5ba49920867"}
Sep 10 03:00:03.469587 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:03.469478    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"
Sep 10 03:00:03.472824 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:03.472803    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"
Sep 10 03:00:03.551313 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:03.551253    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"
Sep 10 03:00:03.554636 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:03.554617    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq"
Sep 10 03:00:04.307103 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.307067    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"
Sep 10 03:00:04.307103 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.307093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz" event={"ID":"029d658e-d8c2-47f4-87b8-1fb69731cd43","Type":"ContainerDied","Data":"ef5371c09110ccb632b982a878e8acf0b1303056f52c2de1ff7bd1ec54e6d96b"}
Sep 10 03:00:04.307378 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.307134    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef5371c09110ccb632b982a878e8acf0b1303056f52c2de1ff7bd1ec54e6d96b"
Sep 10 03:00:04.308332 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.308313    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"
Sep 10 03:00:04.308439 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.308354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8" event={"ID":"468ec338-2565-4c37-8616-8a75622b08d7","Type":"ContainerDied","Data":"4be6aafc03d60cda2c6cfeb01cf49acb7f22f810b0cda2f5b8184af0d2e6a921"}
Sep 10 03:00:04.308439 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.308390    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4be6aafc03d60cda2c6cfeb01cf49acb7f22f810b0cda2f5b8184af0d2e6a921"
Sep 10 03:00:04.309664 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.309645    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq"
Sep 10 03:00:04.309789 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.309641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816820-6mssq" event={"ID":"a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef","Type":"ContainerDied","Data":"0a9badeac6753cb649c01d392e8a467e3090c3f96f706587bb35a1e5f0522b44"}
Sep 10 03:00:04.309789 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.309707    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9badeac6753cb649c01d392e8a467e3090c3f96f706587bb35a1e5f0522b44"
Sep 10 03:00:04.311234 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.311210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p" event={"ID":"adf32dce-3382-48c2-baec-fca97d1a866e","Type":"ContainerDied","Data":"98359651b7da8b7df74d1910846a5d3f57e170b0ef03f28c90ad8c309604b9f7"}
Sep 10 03:00:04.311334 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.311244    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"
Sep 10 03:00:04.311334 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:04.311252    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98359651b7da8b7df74d1910846a5d3f57e170b0ef03f28c90ad8c309604b9f7"
Sep 10 03:00:09.514617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:09.514543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:00:16.717508 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717473    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"]
Sep 10 03:00:16.717877 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717866    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029d658e-d8c2-47f4-87b8-1fb69731cd43" containerName="cleanup"
Sep 10 03:00:16.717919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717879    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="029d658e-d8c2-47f4-87b8-1fb69731cd43" containerName="cleanup"
Sep 10 03:00:16.717919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="468ec338-2565-4c37-8616-8a75622b08d7" containerName="cleanup"
Sep 10 03:00:16.717919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717897    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="468ec338-2565-4c37-8616-8a75622b08d7" containerName="cleanup"
Sep 10 03:00:16.717919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717914    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf32dce-3382-48c2-baec-fca97d1a866e" containerName="cleanup"
Sep 10 03:00:16.717919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717920    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf32dce-3382-48c2-baec-fca97d1a866e" containerName="cleanup"
Sep 10 03:00:16.718080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717933    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef" containerName="cleanup"
Sep 10 03:00:16.718080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717938    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef" containerName="cleanup"
Sep 10 03:00:16.718080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.717996    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="029d658e-d8c2-47f4-87b8-1fb69731cd43" containerName="cleanup"
Sep 10 03:00:16.718080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.718003    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="468ec338-2565-4c37-8616-8a75622b08d7" containerName="cleanup"
Sep 10 03:00:16.718080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.718019    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf32dce-3382-48c2-baec-fca97d1a866e" containerName="cleanup"
Sep 10 03:00:16.718080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.718028    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef" containerName="cleanup"
Sep 10 03:00:16.721106 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.721091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"
Sep 10 03:00:16.725392 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.725132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:00:16.725392 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.725146    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:00:16.728381 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.728355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"]
Sep 10 03:00:16.871443 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.871409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbd8a23-301b-4ec3-a3dc-a9880dfb0311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728\" (UID: \"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"
Sep 10 03:00:16.973022 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.972933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbd8a23-301b-4ec3-a3dc-a9880dfb0311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728\" (UID: \"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"
Sep 10 03:00:16.973587 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:16.973565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbd8a23-301b-4ec3-a3dc-a9880dfb0311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728\" (UID: \"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"
Sep 10 03:00:17.034175 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:17.034134    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"
Sep 10 03:00:17.303578 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:17.303541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"]
Sep 10 03:00:17.305056 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:17.305030    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbd8a23_301b_4ec3_a3dc_a9880dfb0311.slice/crio-ed52f51cef703734ae18b6abeb1bb3c05711f1094c1e3f530060df09f0f0975b WatchSource:0}: Error finding container ed52f51cef703734ae18b6abeb1bb3c05711f1094c1e3f530060df09f0f0975b: Status 404 returned error can't find the container with id ed52f51cef703734ae18b6abeb1bb3c05711f1094c1e3f530060df09f0f0975b
Sep 10 03:00:17.384955 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:17.384919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728" event={"ID":"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311","Type":"ContainerStarted","Data":"ed52f51cef703734ae18b6abeb1bb3c05711f1094c1e3f530060df09f0f0975b"}
Sep 10 03:00:18.392023 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:18.391988    2580 generic.go:356] "Generic (PLEG): container finished" podID="bdbd8a23-301b-4ec3-a3dc-a9880dfb0311" containerID="4df2488e76b471fd2c404138c2ab6f9cadd37754aae3c0d7c6cc2b4086428c38" exitCode=0
Sep 10 03:00:18.392444 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:18.392075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728" event={"ID":"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311","Type":"ContainerDied","Data":"4df2488e76b471fd2c404138c2ab6f9cadd37754aae3c0d7c6cc2b4086428c38"}
Sep 10 03:00:19.524944 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:19.524922    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"
Sep 10 03:00:19.598055 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:19.598028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbd8a23-301b-4ec3-a3dc-a9880dfb0311-maas-api-service-ca\") pod \"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311\" (UID: \"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311\") "
Sep 10 03:00:19.598383 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:19.598362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbd8a23-301b-4ec3-a3dc-a9880dfb0311-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdbd8a23-301b-4ec3-a3dc-a9880dfb0311" (UID: "bdbd8a23-301b-4ec3-a3dc-a9880dfb0311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:00:19.699735 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:19.699675    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbd8a23-301b-4ec3-a3dc-a9880dfb0311-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:00:20.404466 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:20.404437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"
Sep 10 03:00:20.404625 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:20.404440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728" event={"ID":"bdbd8a23-301b-4ec3-a3dc-a9880dfb0311","Type":"ContainerDied","Data":"ed52f51cef703734ae18b6abeb1bb3c05711f1094c1e3f530060df09f0f0975b"}
Sep 10 03:00:20.404625 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:20.404568    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed52f51cef703734ae18b6abeb1bb3c05711f1094c1e3f530060df09f0f0975b"
Sep 10 03:00:23.228203 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:23.228147    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"]
Sep 10 03:00:23.230028 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:23.229999    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-387ddcaa-lb728"]
Sep 10 03:00:23.333585 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:23.333548    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdbd8a23-301b-4ec3-a3dc-a9880dfb0311" path="/var/lib/kubelet/pods/bdbd8a23-301b-4ec3-a3dc-a9880dfb0311/volumes"
Sep 10 03:00:24.132440 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.132406    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"]
Sep 10 03:00:24.132880 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.132863    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbd8a23-301b-4ec3-a3dc-a9880dfb0311" containerName="revoke-keys"
Sep 10 03:00:24.132880 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.132880    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbd8a23-301b-4ec3-a3dc-a9880dfb0311" containerName="revoke-keys"
Sep 10 03:00:24.133031 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.132950    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdbd8a23-301b-4ec3-a3dc-a9880dfb0311" containerName="revoke-keys"
Sep 10 03:00:24.136197 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.136159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.138961 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.138940    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-30dae5eb-openshift-default-dockercfg-ngf2h\""
Sep 10 03:00:24.145096 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.145074    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"]
Sep 10 03:00:24.242719 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.242671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-credential-socket\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243312 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-istio-token\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-socket\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-data\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cfb2af-2069-4c29-9317-849b0783c582-istio-podinfo\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-certs\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243668 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8ksc\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-kube-api-access-x8ksc\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243668 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cfb2af-2069-4c29-9317-849b0783c582-istiod-ca-cert\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.243668 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.243554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-envoy\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344295 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cfb2af-2069-4c29-9317-849b0783c582-istiod-ca-cert\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344493 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-envoy\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344493 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-credential-socket\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344493 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-istio-token\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344670 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-socket\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344670 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-data\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344670 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cfb2af-2069-4c29-9317-849b0783c582-istio-podinfo\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344670 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-certs\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344670 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8ksc\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-kube-api-access-x8ksc\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.344935 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-data\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.345000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.344963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-certs\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.345061 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.345026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-credential-socket\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.345099 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.345074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cfb2af-2069-4c29-9317-849b0783c582-istiod-ca-cert\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.345145 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.345135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-socket\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.347259 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.347239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-envoy\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.347405 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.347388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cfb2af-2069-4c29-9317-849b0783c582-istio-podinfo\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.354467 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.354442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-istio-token\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.354974 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.354955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8ksc\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-kube-api-access-x8ksc\") pod \"e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") " pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.449542 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.449462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:24.723323 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.723292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"]
Sep 10 03:00:24.724418 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:24.724393    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cfb2af_2069_4c29_9317_849b0783c582.slice/crio-ce60a6337b0834bc1e65565be94c8a527b6f2a363e51f2b711f68662d7be0c62 WatchSource:0}: Error finding container ce60a6337b0834bc1e65565be94c8a527b6f2a363e51f2b711f68662d7be0c62: Status 404 returned error can't find the container with id ce60a6337b0834bc1e65565be94c8a527b6f2a363e51f2b711f68662d7be0c62
Sep 10 03:00:24.726768 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.726736    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:00:24.726852 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.726800    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:00:24.726852 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:24.726829    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:00:25.435374 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:25.435341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" event={"ID":"57cfb2af-2069-4c29-9317-849b0783c582","Type":"ContainerStarted","Data":"41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709"}
Sep 10 03:00:25.435740 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:25.435393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" event={"ID":"57cfb2af-2069-4c29-9317-849b0783c582","Type":"ContainerStarted","Data":"ce60a6337b0834bc1e65565be94c8a527b6f2a363e51f2b711f68662d7be0c62"}
Sep 10 03:00:25.450439 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:25.450415    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:25.456536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:25.456510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:25.458145 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:25.458104    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" podStartSLOduration=1.458091822 podStartE2EDuration="1.458091822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:25.454820505 +0000 UTC m=+822.604720202" watchObservedRunningTime="2026-09-10 03:00:25.458091822 +0000 UTC m=+822.607991518"
Sep 10 03:00:26.446925 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:26.446882    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:26.447891 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:26.447871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:28.653451 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.653417    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"]
Sep 10 03:00:28.657374 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.657359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:28.660214 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.660194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-30dae5eb\""
Sep 10 03:00:28.660334 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.660198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-30dae5eb-dockercfg-2n7jc\""
Sep 10 03:00:28.668150 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.668127    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"]
Sep 10 03:00:28.784187 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.783636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bq26\" (UniqueName: \"kubernetes.io/projected/21297c4c-5d88-4753-bbe9-ee46c34af8f5-kube-api-access-9bq26\") pod \"payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:28.784187 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.783741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21297c4c-5d88-4753-bbe9-ee46c34af8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:28.885210 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.885175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bq26\" (UniqueName: \"kubernetes.io/projected/21297c4c-5d88-4753-bbe9-ee46c34af8f5-kube-api-access-9bq26\") pod \"payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:28.885394 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.885332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21297c4c-5d88-4753-bbe9-ee46c34af8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:28.885992 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.885973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21297c4c-5d88-4753-bbe9-ee46c34af8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:28.893034 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.893003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bq26\" (UniqueName: \"kubernetes.io/projected/21297c4c-5d88-4753-bbe9-ee46c34af8f5-kube-api-access-9bq26\") pod \"payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:28.969136 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:28.969059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:29.241984 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:29.241956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"]
Sep 10 03:00:29.243212 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:29.243159    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21297c4c_5d88_4753_bbe9_ee46c34af8f5.slice/crio-24b0cd345f79f64a175f246b702d202818fdf9e299c659591ca0f198bc6c9433 WatchSource:0}: Error finding container 24b0cd345f79f64a175f246b702d202818fdf9e299c659591ca0f198bc6c9433: Status 404 returned error can't find the container with id 24b0cd345f79f64a175f246b702d202818fdf9e299c659591ca0f198bc6c9433
Sep 10 03:00:29.463236 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:29.463198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp" event={"ID":"21297c4c-5d88-4753-bbe9-ee46c34af8f5","Type":"ContainerStarted","Data":"47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda"}
Sep 10 03:00:29.463407 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:29.463258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp" event={"ID":"21297c4c-5d88-4753-bbe9-ee46c34af8f5","Type":"ContainerStarted","Data":"24b0cd345f79f64a175f246b702d202818fdf9e299c659591ca0f198bc6c9433"}
Sep 10 03:00:29.463407 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:29.463286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:29.479885 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:29.479836    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp" podStartSLOduration=1.479816694 podStartE2EDuration="1.479816694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:29.478871913 +0000 UTC m=+826.628771609" watchObservedRunningTime="2026-09-10 03:00:29.479816694 +0000 UTC m=+826.629716413"
Sep 10 03:00:30.211475 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:30.211443    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:00:40.472380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:40.472347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:43.371705 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:43.371673    2580 scope.go:117] "RemoveContainer" containerID="01cd95c4e0ce71629da5a3f4bbd2a5ba9e97b7ea1bdcabf5a0d6db2413157e49"
Sep 10 03:00:44.806754 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.806719    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"]
Sep 10 03:00:44.830189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.824570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:44.830667 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.830426    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"]
Sep 10 03:00:44.841857 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.841445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5ab28b\"/\"default-dockercfg-jswcb\""
Sep 10 03:00:44.841857 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.841691    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5ab28b\"/\"openshift-service-ca.crt\""
Sep 10 03:00:44.842083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.841976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5ab28b\"/\"autores-model-5ab28b-kserve-self-signed-certs\""
Sep 10 03:00:44.842282 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.842257    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5ab28b\"/\"kube-root-ca.crt\""
Sep 10 03:00:44.942297 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.942257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tls-certs\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:44.942472 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.942305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwj4\" (UniqueName: \"kubernetes.io/projected/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-kube-api-access-lnwj4\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:44.942472 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.942411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-home\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:44.942472 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.942442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tmp-dir\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:44.942586 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.942540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-dshm\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:44.942586 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:44.942578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-model-cache\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.043779 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.043730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-home\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.043984 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.043793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tmp-dir\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.043984 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.043883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-dshm\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.043984 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.043937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-model-cache\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.044180 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.044002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tls-certs\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.044180 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.044040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwj4\" (UniqueName: \"kubernetes.io/projected/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-kube-api-access-lnwj4\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.044856 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.044826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-home\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.045121 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.045098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tmp-dir\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.045690 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.045663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-model-cache\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.050471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.050399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-dshm\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.051567 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.051547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tls-certs\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.054372 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.054350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwj4\" (UniqueName: \"kubernetes.io/projected/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-kube-api-access-lnwj4\") pod \"autores-model-5ab28b-kserve-74cf77f979-5jw2v\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") " pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.147042 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.147002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:45.422544 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.420989    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"]
Sep 10 03:00:45.426299 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:00:45.426271    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf3c8680_cad8_4874_9c6d_a7171ffbb6cd.slice/crio-d10c631227d5980601e9283ebaea69d876fd1ac266c6ef011d02757a0cb22238 WatchSource:0}: Error finding container d10c631227d5980601e9283ebaea69d876fd1ac266c6ef011d02757a0cb22238: Status 404 returned error can't find the container with id d10c631227d5980601e9283ebaea69d876fd1ac266c6ef011d02757a0cb22238
Sep 10 03:00:45.561592 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.561553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v" event={"ID":"df3c8680-cad8-4874-9c6d-a7171ffbb6cd","Type":"ContainerStarted","Data":"5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174"}
Sep 10 03:00:45.561759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.561615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v" event={"ID":"df3c8680-cad8-4874-9c6d-a7171ffbb6cd","Type":"ContainerStarted","Data":"d10c631227d5980601e9283ebaea69d876fd1ac266c6ef011d02757a0cb22238"}
Sep 10 03:00:45.580196 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:45.580134    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v" podStartSLOduration=1.580120832 podStartE2EDuration="1.580120832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:45.576919098 +0000 UTC m=+842.726818794" watchObservedRunningTime="2026-09-10 03:00:45.580120832 +0000 UTC m=+842.730020528"
Sep 10 03:00:46.810219 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:46.810180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:00:49.587439 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:49.587408    2580 generic.go:356] "Generic (PLEG): container finished" podID="57cfb2af-2069-4c29-9317-849b0783c582" containerID="41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709" exitCode=0
Sep 10 03:00:49.587829 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:49.587490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" event={"ID":"57cfb2af-2069-4c29-9317-849b0783c582","Type":"ContainerDied","Data":"41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709"}
Sep 10 03:00:49.587886 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:49.587871    2580 scope.go:117] "RemoveContainer" containerID="41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709"
Sep 10 03:00:49.588995 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:49.588971    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:00:49.589065 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:49.589052    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:00:49.589105 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:49.589087    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:00:50.595511 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:50.595477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" event={"ID":"57cfb2af-2069-4c29-9317-849b0783c582","Type":"ContainerStarted","Data":"b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9"}
Sep 10 03:00:51.449788 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:51.449753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:51.449991 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:51.449948    2580 patch_prober.go:28] interesting pod/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 03:00:51.450080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:51.450026    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 03:00:52.450606 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:52.450572    2580 patch_prober.go:28] interesting pod/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 03:00:52.450999 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:52.450639    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 03:00:53.450458 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:53.450423    2580 patch_prober.go:28] interesting pod/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 03:00:53.450636 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:53.450487    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 03:00:54.449932 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:54.449893    2580 patch_prober.go:28] interesting pod/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 03:00:54.450148 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:54.449963    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 03:00:55.147609 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:55.147566    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:55.148105 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:55.147645    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:55.159959 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:55.159936    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:55.450942 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:55.450822    2580 patch_prober.go:28] interesting pod/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 03:00:55.450942 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:55.450880    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 03:00:55.652011 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:55.651981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:00:56.447824 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:56.447794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:56.454135 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:56.454113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:56.647079 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:56.647055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:00:58.621228 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:58.621195    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"]
Sep 10 03:00:58.621670 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:58.621437    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp" podUID="21297c4c-5d88-4753-bbe9-ee46c34af8f5" containerName="payload-pre-processing" containerID="cri-o://47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda" gracePeriod=30
Sep 10 03:00:58.866383 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:58.866360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:58.998847 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:58.998749    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21297c4c-5d88-4753-bbe9-ee46c34af8f5-plugins-config-volume\") pod \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") "
Sep 10 03:00:58.999022 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:58.998886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bq26\" (UniqueName: \"kubernetes.io/projected/21297c4c-5d88-4753-bbe9-ee46c34af8f5-kube-api-access-9bq26\") pod \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\" (UID: \"21297c4c-5d88-4753-bbe9-ee46c34af8f5\") "
Sep 10 03:00:58.999200 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:58.999156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21297c4c-5d88-4753-bbe9-ee46c34af8f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21297c4c-5d88-4753-bbe9-ee46c34af8f5" (UID: "21297c4c-5d88-4753-bbe9-ee46c34af8f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:00:58.999353 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:58.999334    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21297c4c-5d88-4753-bbe9-ee46c34af8f5-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:00:59.000977 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.000946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21297c4c-5d88-4753-bbe9-ee46c34af8f5-kube-api-access-9bq26" (OuterVolumeSpecName: "kube-api-access-9bq26") pod "21297c4c-5d88-4753-bbe9-ee46c34af8f5" (UID: "21297c4c-5d88-4753-bbe9-ee46c34af8f5"). InnerVolumeSpecName "kube-api-access-9bq26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:00:59.099925 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.099893    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bq26\" (UniqueName: \"kubernetes.io/projected/21297c4c-5d88-4753-bbe9-ee46c34af8f5-kube-api-access-9bq26\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:00:59.663288 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.663253    2580 generic.go:356] "Generic (PLEG): container finished" podID="21297c4c-5d88-4753-bbe9-ee46c34af8f5" containerID="47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda" exitCode=0
Sep 10 03:00:59.663757 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.663290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp" event={"ID":"21297c4c-5d88-4753-bbe9-ee46c34af8f5","Type":"ContainerDied","Data":"47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda"}
Sep 10 03:00:59.663757 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.663303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"
Sep 10 03:00:59.663757 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.663354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp" event={"ID":"21297c4c-5d88-4753-bbe9-ee46c34af8f5","Type":"ContainerDied","Data":"24b0cd345f79f64a175f246b702d202818fdf9e299c659591ca0f198bc6c9433"}
Sep 10 03:00:59.663757 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.663389    2580 scope.go:117] "RemoveContainer" containerID="47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda"
Sep 10 03:00:59.672509 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.672487    2580 scope.go:117] "RemoveContainer" containerID="47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda"
Sep 10 03:00:59.672769 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:00:59.672752    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda\": container with ID starting with 47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda not found: ID does not exist" containerID="47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda"
Sep 10 03:00:59.672820 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.672782    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda"} err="failed to get container status \"47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda\": rpc error: code = NotFound desc = could not find container \"47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda\": container with ID starting with 47a42dbf4303f074a3e8738d9950aa2e726931dd6d5e6f466ec60c0a999f8fda not found: ID does not exist"
Sep 10 03:00:59.680143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.680122    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"]
Sep 10 03:00:59.683971 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.683949    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30dae5eb-7c88f68dc4-45qtp"]
Sep 10 03:00:59.994607 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.994526    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"]
Sep 10 03:00:59.994767 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:00:59.994717    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy" containerID="cri-o://b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9" gracePeriod=30
Sep 10 03:01:01.325892 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:01.325853    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21297c4c-5d88-4753-bbe9-ee46c34af8f5" path="/var/lib/kubelet/pods/21297c4c-5d88-4753-bbe9-ee46c34af8f5/volumes"
Sep 10 03:01:03.918227 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:03.918195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:01:05.241284 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.241259    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:01:05.361104 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cfb2af-2069-4c29-9317-849b0783c582-istio-podinfo\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361275 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-credential-socket\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361275 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-data\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361353 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8ksc\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-kube-api-access-x8ksc\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361353 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-envoy\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361425 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:05.361425 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-istio-token\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361531 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361496    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-data" (OuterVolumeSpecName: "istio-data") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:05.361531 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-socket\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361630 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-certs\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361630 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cfb2af-2069-4c29-9317-849b0783c582-istiod-ca-cert\") pod \"57cfb2af-2069-4c29-9317-849b0783c582\" (UID: \"57cfb2af-2069-4c29-9317-849b0783c582\") "
Sep 10 03:01:05.361732 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:05.361800 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:05.361987 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.361961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cfb2af-2069-4c29-9317-849b0783c582-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:01:05.362083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.362065    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.362083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.362077    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.362213 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.362087    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.362213 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.362096    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.362213 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.362103    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cfb2af-2069-4c29-9317-849b0783c582-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.363302 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.363284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57cfb2af-2069-4c29-9317-849b0783c582-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:01:05.363607 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.363587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-istio-token" (OuterVolumeSpecName: "istio-token") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:01:05.363777 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.363761    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:05.363819 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.363800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-kube-api-access-x8ksc" (OuterVolumeSpecName: "kube-api-access-x8ksc") pod "57cfb2af-2069-4c29-9317-849b0783c582" (UID: "57cfb2af-2069-4c29-9317-849b0783c582"). InnerVolumeSpecName "kube-api-access-x8ksc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:01:05.462812 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.462777    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8ksc\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-kube-api-access-x8ksc\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.462812 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.462809    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cfb2af-2069-4c29-9317-849b0783c582-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.462812 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.462818    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cfb2af-2069-4c29-9317-849b0783c582-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.462812 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.462825    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cfb2af-2069-4c29-9317-849b0783c582-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:05.702355 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.702272    2580 generic.go:356] "Generic (PLEG): container finished" podID="57cfb2af-2069-4c29-9317-849b0783c582" containerID="b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9" exitCode=0
Sep 10 03:01:05.702355 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.702318    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"
Sep 10 03:01:05.702536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.702350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" event={"ID":"57cfb2af-2069-4c29-9317-849b0783c582","Type":"ContainerDied","Data":"b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9"}
Sep 10 03:01:05.702536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.702405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642" event={"ID":"57cfb2af-2069-4c29-9317-849b0783c582","Type":"ContainerDied","Data":"ce60a6337b0834bc1e65565be94c8a527b6f2a363e51f2b711f68662d7be0c62"}
Sep 10 03:01:05.702536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.702430    2580 scope.go:117] "RemoveContainer" containerID="b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9"
Sep 10 03:01:05.711957 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.711940    2580 scope.go:117] "RemoveContainer" containerID="41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709"
Sep 10 03:01:05.726445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.726420    2580 scope.go:117] "RemoveContainer" containerID="b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9"
Sep 10 03:01:05.726737 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:01:05.726718    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9\": container with ID starting with b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9 not found: ID does not exist" containerID="b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9"
Sep 10 03:01:05.726786 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.726745    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9"} err="failed to get container status \"b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9\": rpc error: code = NotFound desc = could not find container \"b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9\": container with ID starting with b9570bf169827725c0a42761b56f3e68feb018d847a553e0211d90d7edd6a3e9 not found: ID does not exist"
Sep 10 03:01:05.726786 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.726762    2580 scope.go:117] "RemoveContainer" containerID="41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709"
Sep 10 03:01:05.727032 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:01:05.727013    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709\": container with ID starting with 41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709 not found: ID does not exist" containerID="41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709"
Sep 10 03:01:05.727080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.727037    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709"} err="failed to get container status \"41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709\": rpc error: code = NotFound desc = could not find container \"41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709\": container with ID starting with 41a5990815a9d1676db954ceb220841acdab6a80df2e24100aa02bf3dbf21709 not found: ID does not exist"
Sep 10 03:01:05.731961 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.731940    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"]
Sep 10 03:01:05.735782 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:05.735760    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-30dae5eb-openshift-default-5475dbf994-fj642"]
Sep 10 03:01:07.325737 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:07.325702    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57cfb2af-2069-4c29-9317-849b0783c582" path="/var/lib/kubelet/pods/57cfb2af-2069-4c29-9317-849b0783c582/volumes"
Sep 10 03:01:19.512245 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.512205    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"]
Sep 10 03:01:19.512878 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.512857    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21297c4c-5d88-4753-bbe9-ee46c34af8f5" containerName="payload-pre-processing"
Sep 10 03:01:19.512953 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.512881    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21297c4c-5d88-4753-bbe9-ee46c34af8f5" containerName="payload-pre-processing"
Sep 10 03:01:19.512953 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.512905    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy"
Sep 10 03:01:19.512953 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.512914    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy"
Sep 10 03:01:19.512953 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.512934    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy"
Sep 10 03:01:19.512953 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.512943    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy"
Sep 10 03:01:19.513127 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.513066    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy"
Sep 10 03:01:19.513127 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.513076    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21297c4c-5d88-4753-bbe9-ee46c34af8f5" containerName="payload-pre-processing"
Sep 10 03:01:19.515883 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.515862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.518770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.518746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0f4da\"/\"kube-root-ca.crt\""
Sep 10 03:01:19.518770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.518772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b0f4da\"/\"openshift-service-ca.crt\""
Sep 10 03:01:19.520203 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.520144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0f4da\"/\"default-dockercfg-rgrkb\""
Sep 10 03:01:19.520382 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.520256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b0f4da\"/\"test-model-b0f4da-kserve-self-signed-certs\""
Sep 10 03:01:19.523743 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.523694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"]
Sep 10 03:01:19.586771 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.586726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-model-cache\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.586771 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.586773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tmp-dir\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.587025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.586836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-home\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.587025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.586888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-dshm\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.587025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.586962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tls-certs\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.587025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.586991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dqk\" (UniqueName: \"kubernetes.io/projected/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-kube-api-access-r5dqk\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688395 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dqk\" (UniqueName: \"kubernetes.io/projected/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-kube-api-access-r5dqk\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-model-cache\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tmp-dir\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-home\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-dshm\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688730 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tls-certs\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-model-cache\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688946 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-home\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.688946 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.688936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tmp-dir\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.690836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.690815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-dshm\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.691084 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.691069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tls-certs\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.697112 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.697086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dqk\" (UniqueName: \"kubernetes.io/projected/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-kube-api-access-r5dqk\") pod \"test-model-b0f4da-kserve-778c74d86c-9g8pm\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") " pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:19.826462 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:19.826381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:20.096905 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.096879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"]
Sep 10 03:01:20.098466 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:01:20.098436    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95bd4ab9_5d5d_4c36_b14a_50d875c5cb1c.slice/crio-5532544d4b677d17a7a2735065d1690cd7d71e3d85099495f4d4d04e55559cec WatchSource:0}: Error finding container 5532544d4b677d17a7a2735065d1690cd7d71e3d85099495f4d4d04e55559cec: Status 404 returned error can't find the container with id 5532544d4b677d17a7a2735065d1690cd7d71e3d85099495f4d4d04e55559cec
Sep 10 03:01:20.510635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.510606    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"]
Sep 10 03:01:20.511102 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.511089    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cfb2af-2069-4c29-9317-849b0783c582" containerName="istio-proxy"
Sep 10 03:01:20.514191 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.514155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"
Sep 10 03:01:20.516977 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.516958    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:01:20.517098 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.516958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:01:20.521779 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.521643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"]
Sep 10 03:01:20.597615 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.597578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e0573e-1c60-4557-9bb5-9c20d7ec8646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs\" (UID: \"16e0573e-1c60-4557-9bb5-9c20d7ec8646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"
Sep 10 03:01:20.698815 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.698785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e0573e-1c60-4557-9bb5-9c20d7ec8646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs\" (UID: \"16e0573e-1c60-4557-9bb5-9c20d7ec8646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"
Sep 10 03:01:20.699422 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.699404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e0573e-1c60-4557-9bb5-9c20d7ec8646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs\" (UID: \"16e0573e-1c60-4557-9bb5-9c20d7ec8646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"
Sep 10 03:01:20.786336 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.786236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm" event={"ID":"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c","Type":"ContainerStarted","Data":"a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7"}
Sep 10 03:01:20.786336 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.786285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm" event={"ID":"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c","Type":"ContainerStarted","Data":"5532544d4b677d17a7a2735065d1690cd7d71e3d85099495f4d4d04e55559cec"}
Sep 10 03:01:20.808143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.808097    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm" podStartSLOduration=1.8080820370000001 podStartE2EDuration="1.808082037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:20.80575657 +0000 UTC m=+877.955656267" watchObservedRunningTime="2026-09-10 03:01:20.808082037 +0000 UTC m=+877.957981735"
Sep 10 03:01:20.825513 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:20.825484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"
Sep 10 03:01:21.096147 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:21.096116    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"]
Sep 10 03:01:21.097362 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:01:21.097338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e0573e_1c60_4557_9bb5_9c20d7ec8646.slice/crio-16ebe5489816474710a6b73cd40966a7c6c580235431ae32079334f1a905e781 WatchSource:0}: Error finding container 16ebe5489816474710a6b73cd40966a7c6c580235431ae32079334f1a905e781: Status 404 returned error can't find the container with id 16ebe5489816474710a6b73cd40966a7c6c580235431ae32079334f1a905e781
Sep 10 03:01:21.793080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:21.792990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs" event={"ID":"16e0573e-1c60-4557-9bb5-9c20d7ec8646","Type":"ContainerStarted","Data":"ba16bbe54f678882660a0e9f443cf9a0fdbd4b089840ae5d720264604cc63055"}
Sep 10 03:01:21.793080 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:21.793040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs" event={"ID":"16e0573e-1c60-4557-9bb5-9c20d7ec8646","Type":"ContainerStarted","Data":"16ebe5489816474710a6b73cd40966a7c6c580235431ae32079334f1a905e781"}
Sep 10 03:01:21.808495 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:21.808451    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs" podStartSLOduration=1.808437069 podStartE2EDuration="1.808437069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:21.807437015 +0000 UTC m=+878.957336714" watchObservedRunningTime="2026-09-10 03:01:21.808437069 +0000 UTC m=+878.958336765"
Sep 10 03:01:22.800218 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:22.800159    2580 generic.go:356] "Generic (PLEG): container finished" podID="16e0573e-1c60-4557-9bb5-9c20d7ec8646" containerID="ba16bbe54f678882660a0e9f443cf9a0fdbd4b089840ae5d720264604cc63055" exitCode=0
Sep 10 03:01:22.800613 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:22.800275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs" event={"ID":"16e0573e-1c60-4557-9bb5-9c20d7ec8646","Type":"ContainerDied","Data":"ba16bbe54f678882660a0e9f443cf9a0fdbd4b089840ae5d720264604cc63055"}
Sep 10 03:01:23.938683 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:23.938660    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"
Sep 10 03:01:24.029048 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:24.029018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e0573e-1c60-4557-9bb5-9c20d7ec8646-maas-api-service-ca\") pod \"16e0573e-1c60-4557-9bb5-9c20d7ec8646\" (UID: \"16e0573e-1c60-4557-9bb5-9c20d7ec8646\") "
Sep 10 03:01:24.029395 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:24.029372    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e0573e-1c60-4557-9bb5-9c20d7ec8646-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16e0573e-1c60-4557-9bb5-9c20d7ec8646" (UID: "16e0573e-1c60-4557-9bb5-9c20d7ec8646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:01:24.130158 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:24.130131    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e0573e-1c60-4557-9bb5-9c20d7ec8646-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:24.812697 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:24.812664    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"
Sep 10 03:01:24.812900 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:24.812721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs" event={"ID":"16e0573e-1c60-4557-9bb5-9c20d7ec8646","Type":"ContainerDied","Data":"16ebe5489816474710a6b73cd40966a7c6c580235431ae32079334f1a905e781"}
Sep 10 03:01:24.812900 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:24.812779    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16ebe5489816474710a6b73cd40966a7c6c580235431ae32079334f1a905e781"
Sep 10 03:01:24.816461 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:24.816434    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:01:27.899111 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.899077    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"]
Sep 10 03:01:27.899527 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.899514    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e0573e-1c60-4557-9bb5-9c20d7ec8646" containerName="revoke-keys"
Sep 10 03:01:27.899527 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.899528    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e0573e-1c60-4557-9bb5-9c20d7ec8646" containerName="revoke-keys"
Sep 10 03:01:27.899612 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.899596    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e0573e-1c60-4557-9bb5-9c20d7ec8646" containerName="revoke-keys"
Sep 10 03:01:27.902692 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.902677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"
Sep 10 03:01:27.905299 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.905274    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:01:27.905445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.905427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:01:27.909911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.909888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"]
Sep 10 03:01:27.973463 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:27.973423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf0558c-bd07-4645-953d-7890d09bb440-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-08482c4c-npphw\" (UID: \"fbf0558c-bd07-4645-953d-7890d09bb440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"
Sep 10 03:01:28.074480 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:28.074453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf0558c-bd07-4645-953d-7890d09bb440-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-08482c4c-npphw\" (UID: \"fbf0558c-bd07-4645-953d-7890d09bb440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"
Sep 10 03:01:28.075071 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:28.075049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf0558c-bd07-4645-953d-7890d09bb440-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-08482c4c-npphw\" (UID: \"fbf0558c-bd07-4645-953d-7890d09bb440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"
Sep 10 03:01:28.215752 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:28.215672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"
Sep 10 03:01:28.483463 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:28.483436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"]
Sep 10 03:01:28.485330 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:01:28.485291    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf0558c_bd07_4645_953d_7890d09bb440.slice/crio-14fcb72c485db5919fe2ee7a8eabea8cc23c6e824128231972c07018b84f70e1 WatchSource:0}: Error finding container 14fcb72c485db5919fe2ee7a8eabea8cc23c6e824128231972c07018b84f70e1: Status 404 returned error can't find the container with id 14fcb72c485db5919fe2ee7a8eabea8cc23c6e824128231972c07018b84f70e1
Sep 10 03:01:28.836338 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:28.836308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw" event={"ID":"fbf0558c-bd07-4645-953d-7890d09bb440","Type":"ContainerStarted","Data":"c007a35043ff04945663c0a4e9324912f4299ce188912c56d97a49faa1ac8235"}
Sep 10 03:01:28.836451 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:28.836356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw" event={"ID":"fbf0558c-bd07-4645-953d-7890d09bb440","Type":"ContainerStarted","Data":"14fcb72c485db5919fe2ee7a8eabea8cc23c6e824128231972c07018b84f70e1"}
Sep 10 03:01:28.852068 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:28.852019    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw" podStartSLOduration=1.8520039860000002 podStartE2EDuration="1.852003986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:28.851100914 +0000 UTC m=+886.001000613" watchObservedRunningTime="2026-09-10 03:01:28.852003986 +0000 UTC m=+886.001903683"
Sep 10 03:01:29.827089 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:29.827056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:29.827491 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:29.827136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:29.839643 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:29.839623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:29.843416 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:29.843394    2580 generic.go:356] "Generic (PLEG): container finished" podID="fbf0558c-bd07-4645-953d-7890d09bb440" containerID="c007a35043ff04945663c0a4e9324912f4299ce188912c56d97a49faa1ac8235" exitCode=0
Sep 10 03:01:29.843550 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:29.843472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw" event={"ID":"fbf0558c-bd07-4645-953d-7890d09bb440","Type":"ContainerDied","Data":"c007a35043ff04945663c0a4e9324912f4299ce188912c56d97a49faa1ac8235"}
Sep 10 03:01:29.854497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:29.854476    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:01:30.046670 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:30.046636    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"]
Sep 10 03:01:30.983536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:30.983500    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"
Sep 10 03:01:31.108328 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.108249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf0558c-bd07-4645-953d-7890d09bb440-maas-api-service-ca\") pod \"fbf0558c-bd07-4645-953d-7890d09bb440\" (UID: \"fbf0558c-bd07-4645-953d-7890d09bb440\") "
Sep 10 03:01:31.108610 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.108588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf0558c-bd07-4645-953d-7890d09bb440-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbf0558c-bd07-4645-953d-7890d09bb440" (UID: "fbf0558c-bd07-4645-953d-7890d09bb440"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:01:31.209589 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.209556    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf0558c-bd07-4645-953d-7890d09bb440-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:31.312306 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.312268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"]
Sep 10 03:01:31.316214 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.316188    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5ab28b-29816820-sw6jz"]
Sep 10 03:01:31.325495 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.325466    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029d658e-d8c2-47f4-87b8-1fb69731cd43" path="/var/lib/kubelet/pods/029d658e-d8c2-47f4-87b8-1fb69731cd43/volumes"
Sep 10 03:01:31.357629 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.357595    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"]
Sep 10 03:01:31.357911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.357882    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p" podUID="f923b25d-2ac4-4b65-96f3-350c7c6602bc" containerName="payload-pre-processing" containerID="cri-o://35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5" gracePeriod=30
Sep 10 03:01:31.567468 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.567394    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"]
Sep 10 03:01:31.572206 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.572175    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5ab28b-qw8vs"]
Sep 10 03:01:31.604944 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.604915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 03:01:31.715593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.715515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc8jx\" (UniqueName: \"kubernetes.io/projected/f923b25d-2ac4-4b65-96f3-350c7c6602bc-kube-api-access-lc8jx\") pod \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") "
Sep 10 03:01:31.715593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.715551    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f923b25d-2ac4-4b65-96f3-350c7c6602bc-plugins-config-volume\") pod \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\" (UID: \"f923b25d-2ac4-4b65-96f3-350c7c6602bc\") "
Sep 10 03:01:31.715918 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.715888    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f923b25d-2ac4-4b65-96f3-350c7c6602bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f923b25d-2ac4-4b65-96f3-350c7c6602bc" (UID: "f923b25d-2ac4-4b65-96f3-350c7c6602bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:01:31.717584 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.717562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f923b25d-2ac4-4b65-96f3-350c7c6602bc-kube-api-access-lc8jx" (OuterVolumeSpecName: "kube-api-access-lc8jx") pod "f923b25d-2ac4-4b65-96f3-350c7c6602bc" (UID: "f923b25d-2ac4-4b65-96f3-350c7c6602bc"). InnerVolumeSpecName "kube-api-access-lc8jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:01:31.816398 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.816365    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc8jx\" (UniqueName: \"kubernetes.io/projected/f923b25d-2ac4-4b65-96f3-350c7c6602bc-kube-api-access-lc8jx\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:31.816398 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.816393    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f923b25d-2ac4-4b65-96f3-350c7c6602bc-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:31.856116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.856086    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"
Sep 10 03:01:31.856313 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.856123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw" event={"ID":"fbf0558c-bd07-4645-953d-7890d09bb440","Type":"ContainerDied","Data":"14fcb72c485db5919fe2ee7a8eabea8cc23c6e824128231972c07018b84f70e1"}
Sep 10 03:01:31.856313 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.856231    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14fcb72c485db5919fe2ee7a8eabea8cc23c6e824128231972c07018b84f70e1"
Sep 10 03:01:31.857549 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.857525    2580 generic.go:356] "Generic (PLEG): container finished" podID="f923b25d-2ac4-4b65-96f3-350c7c6602bc" containerID="35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5" exitCode=0
Sep 10 03:01:31.857651 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.857565    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"
Sep 10 03:01:31.857651 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.857613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p" event={"ID":"f923b25d-2ac4-4b65-96f3-350c7c6602bc","Type":"ContainerDied","Data":"35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5"}
Sep 10 03:01:31.857727 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.857664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p" event={"ID":"f923b25d-2ac4-4b65-96f3-350c7c6602bc","Type":"ContainerDied","Data":"ceb4830cd1584e2da00bb9bb3f06ca0c74c2d975f399e7c01371e78858567fcb"}
Sep 10 03:01:31.857727 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.857696    2580 scope.go:117] "RemoveContainer" containerID="35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5"
Sep 10 03:01:31.866921 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.866906    2580 scope.go:117] "RemoveContainer" containerID="35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5"
Sep 10 03:01:31.867197 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:01:31.867151    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5\": container with ID starting with 35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5 not found: ID does not exist" containerID="35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5"
Sep 10 03:01:31.867284 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.867207    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5"} err="failed to get container status \"35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5\": rpc error: code = NotFound desc = could not find container \"35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5\": container with ID starting with 35c94a64a6286ccb5b234d36b97bf78a171f3bd9985e5dbfac0b58c92fa01fe5 not found: ID does not exist"
Sep 10 03:01:31.878822 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.878776    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"]
Sep 10 03:01:31.880914 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:31.880895    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5ab28b-f84ffd694-5kv6p"]
Sep 10 03:01:33.324983 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:33.324952    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e0573e-1c60-4557-9bb5-9c20d7ec8646" path="/var/lib/kubelet/pods/16e0573e-1c60-4557-9bb5-9c20d7ec8646/volumes"
Sep 10 03:01:33.325359 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:33.325320    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f923b25d-2ac4-4b65-96f3-350c7c6602bc" path="/var/lib/kubelet/pods/f923b25d-2ac4-4b65-96f3-350c7c6602bc/volumes"
Sep 10 03:01:35.034051 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:35.034015    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"]
Sep 10 03:01:35.036574 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:35.036546    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08482c4c-npphw"]
Sep 10 03:01:35.325072 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:35.324991    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbf0558c-bd07-4645-953d-7890d09bb440" path="/var/lib/kubelet/pods/fbf0558c-bd07-4645-953d-7890d09bb440/volumes"
Sep 10 03:01:45.066830 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.066770    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v" podUID="df3c8680-cad8-4874-9c6d-a7171ffbb6cd" containerName="main" containerID="cri-o://5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174" gracePeriod=45
Sep 10 03:01:45.321219 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.321147    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:01:45.456974 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.456944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-model-cache\") pod \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") "
Sep 10 03:01:45.457156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457001    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-dshm\") pod \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") "
Sep 10 03:01:45.457156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457084    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-home\") pod \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") "
Sep 10 03:01:45.457156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tmp-dir\") pod \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") "
Sep 10 03:01:45.457156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnwj4\" (UniqueName: \"kubernetes.io/projected/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-kube-api-access-lnwj4\") pod \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") "
Sep 10 03:01:45.457412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "df3c8680-cad8-4874-9c6d-a7171ffbb6cd" (UID: "df3c8680-cad8-4874-9c6d-a7171ffbb6cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:45.457412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tls-certs\") pod \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\" (UID: \"df3c8680-cad8-4874-9c6d-a7171ffbb6cd\") "
Sep 10 03:01:45.457412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df3c8680-cad8-4874-9c6d-a7171ffbb6cd" (UID: "df3c8680-cad8-4874-9c6d-a7171ffbb6cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:45.457412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457367    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-home" (OuterVolumeSpecName: "home") pod "df3c8680-cad8-4874-9c6d-a7171ffbb6cd" (UID: "df3c8680-cad8-4874-9c6d-a7171ffbb6cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:45.457599 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457560    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:45.457599 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457573    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:45.457697 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.457601    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:45.459243 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.459221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-dshm" (OuterVolumeSpecName: "dshm") pod "df3c8680-cad8-4874-9c6d-a7171ffbb6cd" (UID: "df3c8680-cad8-4874-9c6d-a7171ffbb6cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:01:45.459369 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.459321    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df3c8680-cad8-4874-9c6d-a7171ffbb6cd" (UID: "df3c8680-cad8-4874-9c6d-a7171ffbb6cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:01:45.459369 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.459348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-kube-api-access-lnwj4" (OuterVolumeSpecName: "kube-api-access-lnwj4") pod "df3c8680-cad8-4874-9c6d-a7171ffbb6cd" (UID: "df3c8680-cad8-4874-9c6d-a7171ffbb6cd"). InnerVolumeSpecName "kube-api-access-lnwj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:01:45.558544 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.558515    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:45.558544 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.558540    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwj4\" (UniqueName: \"kubernetes.io/projected/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-kube-api-access-lnwj4\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:45.558686 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.558550    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3c8680-cad8-4874-9c6d-a7171ffbb6cd-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:01:45.936875 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.936838    2580 generic.go:356] "Generic (PLEG): container finished" podID="df3c8680-cad8-4874-9c6d-a7171ffbb6cd" containerID="5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174" exitCode=0
Sep 10 03:01:45.937043 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.936881    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"
Sep 10 03:01:45.937043 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.936918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v" event={"ID":"df3c8680-cad8-4874-9c6d-a7171ffbb6cd","Type":"ContainerDied","Data":"5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174"}
Sep 10 03:01:45.937043 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.936963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v" event={"ID":"df3c8680-cad8-4874-9c6d-a7171ffbb6cd","Type":"ContainerDied","Data":"d10c631227d5980601e9283ebaea69d876fd1ac266c6ef011d02757a0cb22238"}
Sep 10 03:01:45.937043 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.936981    2580 scope.go:117] "RemoveContainer" containerID="5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174"
Sep 10 03:01:45.946146 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.946125    2580 scope.go:117] "RemoveContainer" containerID="5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174"
Sep 10 03:01:45.946407 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:01:45.946388    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174\": container with ID starting with 5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174 not found: ID does not exist" containerID="5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174"
Sep 10 03:01:45.946475 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.946416    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174"} err="failed to get container status \"5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174\": rpc error: code = NotFound desc = could not find container \"5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174\": container with ID starting with 5c2213c36a96c4cc6fbaba275ebe758fdce7e29f46b15bcc2b41789177e81174 not found: ID does not exist"
Sep 10 03:01:45.957967 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.957943    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"]
Sep 10 03:01:45.961335 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:45.961313    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5ab28b/autores-model-5ab28b-kserve-74cf77f979-5jw2v"]
Sep 10 03:01:47.325560 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:47.325524    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df3c8680-cad8-4874-9c6d-a7171ffbb6cd" path="/var/lib/kubelet/pods/df3c8680-cad8-4874-9c6d-a7171ffbb6cd/volumes"
Sep 10 03:01:51.120557 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:01:51.120520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:02:10.310671 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:10.310561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:02:14.136891 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:14.136853    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"]
Sep 10 03:02:14.137391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:14.137102    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy" containerID="cri-o://e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40" gracePeriod=30
Sep 10 03:02:19.386089 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.386057    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 03:02:19.481740 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-data\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.481740 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-credential-socket\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.481740 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481727    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-socket\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.482040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481775    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-certs\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.482040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04541d5c-1857-4df4-8070-00f6e6e34fcf-istiod-ca-cert\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.482040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-envoy\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.482040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-podinfo\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.482040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-data" (OuterVolumeSpecName: "istio-data") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:02:19.482040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481970    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:02:19.482040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.481981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-token\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.482423 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482089    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:02:19.482423 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:02:19.482423 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw59t\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-kube-api-access-mw59t\") pod \"04541d5c-1857-4df4-8070-00f6e6e34fcf\" (UID: \"04541d5c-1857-4df4-8070-00f6e6e34fcf\") "
Sep 10 03:02:19.482423 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04541d5c-1857-4df4-8070-00f6e6e34fcf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:02:19.482637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482570    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.482637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482591    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04541d5c-1857-4df4-8070-00f6e6e34fcf-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.482637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482605    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.482637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482617    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.482637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.482633    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.484087 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.484059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-kube-api-access-mw59t" (OuterVolumeSpecName: "kube-api-access-mw59t") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "kube-api-access-mw59t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:02:19.484225 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.484148    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:02:19.484480 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.484460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:02:19.484541 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.484492    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-token" (OuterVolumeSpecName: "istio-token") pod "04541d5c-1857-4df4-8070-00f6e6e34fcf" (UID: "04541d5c-1857-4df4-8070-00f6e6e34fcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:02:19.583467 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.583439    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.583467 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.583465    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.583467 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.583473    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.583467 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.583482    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw59t\" (UniqueName: \"kubernetes.io/projected/04541d5c-1857-4df4-8070-00f6e6e34fcf-kube-api-access-mw59t\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:02:19.831843 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.831755    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"]
Sep 10 03:02:19.832215 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832202    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf0558c-bd07-4645-953d-7890d09bb440" containerName="revoke-keys"
Sep 10 03:02:19.832274 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832217    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf0558c-bd07-4645-953d-7890d09bb440" containerName="revoke-keys"
Sep 10 03:02:19.832274 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832237    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy"
Sep 10 03:02:19.832274 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832244    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy"
Sep 10 03:02:19.832274 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832269    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f923b25d-2ac4-4b65-96f3-350c7c6602bc" containerName="payload-pre-processing"
Sep 10 03:02:19.832274 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832275    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f923b25d-2ac4-4b65-96f3-350c7c6602bc" containerName="payload-pre-processing"
Sep 10 03:02:19.832432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832282    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3c8680-cad8-4874-9c6d-a7171ffbb6cd" containerName="main"
Sep 10 03:02:19.832432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832287    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3c8680-cad8-4874-9c6d-a7171ffbb6cd" containerName="main"
Sep 10 03:02:19.832432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f923b25d-2ac4-4b65-96f3-350c7c6602bc" containerName="payload-pre-processing"
Sep 10 03:02:19.832432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832360    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf0558c-bd07-4645-953d-7890d09bb440" containerName="revoke-keys"
Sep 10 03:02:19.832432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832368    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy"
Sep 10 03:02:19.832432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.832374    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3c8680-cad8-4874-9c6d-a7171ffbb6cd" containerName="main"
Sep 10 03:02:19.835745 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.835727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:19.838915 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.838894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-af329f-dockercfg-bphph\""
Sep 10 03:02:19.838915 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.838910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-af329f\""
Sep 10 03:02:19.846270 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.846227    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"]
Sep 10 03:02:19.879281 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.879249    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"]
Sep 10 03:02:19.879717 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.879700    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy"
Sep 10 03:02:19.879804 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.879720    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy"
Sep 10 03:02:19.879860 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.879843    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerName="istio-proxy"
Sep 10 03:02:19.882921 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.882903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:19.893684 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.893660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"]
Sep 10 03:02:19.987910 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.987876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-plugins-config-volume\") pod \"payload-processing-e2e-explicit-af329f-6c6c696665-f6x99\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") " pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:19.988079 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.987917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abecd37-49ec-4f63-b2d6-07640f536cec-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:19.988079 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.988039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6gq\" (UniqueName: \"kubernetes.io/projected/7abecd37-49ec-4f63-b2d6-07640f536cec-kube-api-access-5g6gq\") pod \"payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:19.988224 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:19.988100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhxh\" (UniqueName: \"kubernetes.io/projected/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-kube-api-access-4mhxh\") pod \"payload-processing-e2e-explicit-af329f-6c6c696665-f6x99\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") " pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:20.089052 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.088973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6gq\" (UniqueName: \"kubernetes.io/projected/7abecd37-49ec-4f63-b2d6-07640f536cec-kube-api-access-5g6gq\") pod \"payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:20.089052 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.089030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhxh\" (UniqueName: \"kubernetes.io/projected/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-kube-api-access-4mhxh\") pod \"payload-processing-e2e-explicit-af329f-6c6c696665-f6x99\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") " pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:20.089268 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.089079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-plugins-config-volume\") pod \"payload-processing-e2e-explicit-af329f-6c6c696665-f6x99\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") " pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:20.089268 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.089113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abecd37-49ec-4f63-b2d6-07640f536cec-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:20.089811 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.089790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abecd37-49ec-4f63-b2d6-07640f536cec-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:20.089811 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.089805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-plugins-config-volume\") pod \"payload-processing-e2e-explicit-af329f-6c6c696665-f6x99\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") " pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:20.097213 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.097193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhxh\" (UniqueName: \"kubernetes.io/projected/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-kube-api-access-4mhxh\") pod \"payload-processing-e2e-explicit-af329f-6c6c696665-f6x99\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") " pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:20.097335 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.097275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6gq\" (UniqueName: \"kubernetes.io/projected/7abecd37-49ec-4f63-b2d6-07640f536cec-kube-api-access-5g6gq\") pod \"payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:20.134286 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.134249    2580 generic.go:356] "Generic (PLEG): container finished" podID="04541d5c-1857-4df4-8070-00f6e6e34fcf" containerID="e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40" exitCode=0
Sep 10 03:02:20.134396 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.134311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"
Sep 10 03:02:20.134396 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.134314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" event={"ID":"04541d5c-1857-4df4-8070-00f6e6e34fcf","Type":"ContainerDied","Data":"e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40"}
Sep 10 03:02:20.134396 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.134391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf" event={"ID":"04541d5c-1857-4df4-8070-00f6e6e34fcf","Type":"ContainerDied","Data":"cc3983bab0758b6affd99940d7ba7a8e0875cbf16a73628cfdccf48cb4352d2a"}
Sep 10 03:02:20.134510 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.134421    2580 scope.go:117] "RemoveContainer" containerID="e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40"
Sep 10 03:02:20.143478 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.143462    2580 scope.go:117] "RemoveContainer" containerID="ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887"
Sep 10 03:02:20.147513 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.147493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:20.151759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.151739    2580 scope.go:117] "RemoveContainer" containerID="e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40"
Sep 10 03:02:20.152008 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:02:20.151991    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40\": container with ID starting with e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40 not found: ID does not exist" containerID="e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40"
Sep 10 03:02:20.152058 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.152016    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40"} err="failed to get container status \"e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40\": rpc error: code = NotFound desc = could not find container \"e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40\": container with ID starting with e257c07a6648c9d856a9daa4c5a6fe20af71f3daf1409c9c568e952a44ea5f40 not found: ID does not exist"
Sep 10 03:02:20.152058 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.152033    2580 scope.go:117] "RemoveContainer" containerID="ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887"
Sep 10 03:02:20.156131 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:02:20.153643    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887\": container with ID starting with ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887 not found: ID does not exist" containerID="ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887"
Sep 10 03:02:20.156131 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.153674    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887"} err="failed to get container status \"ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887\": rpc error: code = NotFound desc = could not find container \"ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887\": container with ID starting with ecb6dcd4dff96549a743743168d37f7e0f0e280776813b9e7f070fab1a778887 not found: ID does not exist"
Sep 10 03:02:20.157740 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.157716    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"]
Sep 10 03:02:20.159815 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.159793    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5ab28b-openshift-default-766bdbdcfd-45xrf"]
Sep 10 03:02:20.193784 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.193747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:20.422743 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.422716    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"]
Sep 10 03:02:20.423755 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:02:20.423730    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abecd37_49ec_4f63_b2d6_07640f536cec.slice/crio-77e6d9e578142ec03a44bc8d11674ac0d73c9eba8567fc7899f2d1ad0088d0f6 WatchSource:0}: Error finding container 77e6d9e578142ec03a44bc8d11674ac0d73c9eba8567fc7899f2d1ad0088d0f6: Status 404 returned error can't find the container with id 77e6d9e578142ec03a44bc8d11674ac0d73c9eba8567fc7899f2d1ad0088d0f6
Sep 10 03:02:20.480687 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:20.480662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"]
Sep 10 03:02:20.481868 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:02:20.481841    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c5f35d_7b77_49f5_9eaa_df85cdb9af67.slice/crio-ee22ea4491291677e316abb876685f5c2ceee8178b7e2139d414745528a54c23 WatchSource:0}: Error finding container ee22ea4491291677e316abb876685f5c2ceee8178b7e2139d414745528a54c23: Status 404 returned error can't find the container with id ee22ea4491291677e316abb876685f5c2ceee8178b7e2139d414745528a54c23
Sep 10 03:02:21.144134 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.144092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99" event={"ID":"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67","Type":"ContainerStarted","Data":"dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f"}
Sep 10 03:02:21.144335 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.144159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99" event={"ID":"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67","Type":"ContainerStarted","Data":"ee22ea4491291677e316abb876685f5c2ceee8178b7e2139d414745528a54c23"}
Sep 10 03:02:21.144335 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.144201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:21.145795 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.145764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w" event={"ID":"7abecd37-49ec-4f63-b2d6-07640f536cec","Type":"ContainerStarted","Data":"7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d"}
Sep 10 03:02:21.145938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.145814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w" event={"ID":"7abecd37-49ec-4f63-b2d6-07640f536cec","Type":"ContainerStarted","Data":"77e6d9e578142ec03a44bc8d11674ac0d73c9eba8567fc7899f2d1ad0088d0f6"}
Sep 10 03:02:21.145938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.145838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:21.165319 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.165280    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99" podStartSLOduration=2.165267392 podStartE2EDuration="2.165267392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:02:21.16339143 +0000 UTC m=+938.313291130" watchObservedRunningTime="2026-09-10 03:02:21.165267392 +0000 UTC m=+938.315167088"
Sep 10 03:02:21.182387 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.182338    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w" podStartSLOduration=2.182321344 podStartE2EDuration="2.182321344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:02:21.181688884 +0000 UTC m=+938.331588584" watchObservedRunningTime="2026-09-10 03:02:21.182321344 +0000 UTC m=+938.332221043"
Sep 10 03:02:21.325173 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:21.325129    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04541d5c-1857-4df4-8070-00f6e6e34fcf" path="/var/lib/kubelet/pods/04541d5c-1857-4df4-8070-00f6e6e34fcf/volumes"
Sep 10 03:02:30.313504 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:30.313468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:02:31.707580 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.707546    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"]
Sep 10 03:02:31.712232 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.712212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:31.715021 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.715000    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"kube-root-ca.crt\""
Sep 10 03:02:31.715150 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.715028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"default-dockercfg-vbcwd\""
Sep 10 03:02:31.716358 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.716337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"openshift-service-ca.crt\""
Sep 10 03:02:31.716481 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.716376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"premium-w4-e6e6b8-kserve-self-signed-certs\""
Sep 10 03:02:31.721565 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.721541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"]
Sep 10 03:02:31.900320 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.900290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-model-cache\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:31.900497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.900332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-dshm\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:31.900497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.900355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-home\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:31.900497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.900444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tmp-dir\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:31.900497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.900464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tls-certs\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:31.900497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:31.900483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sx9l\" (UniqueName: \"kubernetes.io/projected/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-kube-api-access-8sx9l\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001089 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.000998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-home\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001279 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tmp-dir\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001279 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tls-certs\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sx9l\" (UniqueName: \"kubernetes.io/projected/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-kube-api-access-8sx9l\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001460 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-model-cache\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001460 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-home\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001563 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-dshm\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001563 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tmp-dir\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.001689 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.001669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-model-cache\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.003506 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.003476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-dshm\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.003676 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.003661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tls-certs\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.009445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.009424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sx9l\" (UniqueName: \"kubernetes.io/projected/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-kube-api-access-8sx9l\") pod \"premium-w4-e6e6b8-kserve-54dcf74594-f8449\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") " pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.024315 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.024296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:32.153109 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.153081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:02:32.153264 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.153147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:02:32.292065 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:32.292039    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"]
Sep 10 03:02:32.293992 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:02:32.293969    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69fa6cbb_0c0c_4990_9d6d_ac5b5613ef00.slice/crio-451bd13791427630e098474a2c15cd3814efbc07dd9ab13c6423956de0edf62e WatchSource:0}: Error finding container 451bd13791427630e098474a2c15cd3814efbc07dd9ab13c6423956de0edf62e: Status 404 returned error can't find the container with id 451bd13791427630e098474a2c15cd3814efbc07dd9ab13c6423956de0edf62e
Sep 10 03:02:33.214550 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:33.214517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449" event={"ID":"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00","Type":"ContainerStarted","Data":"e79371ca404e7283aff3d40a4a96fb8624e4e09bd24e9e5f298ebb95ca618f25"}
Sep 10 03:02:33.214909 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:33.214564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449" event={"ID":"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00","Type":"ContainerStarted","Data":"451bd13791427630e098474a2c15cd3814efbc07dd9ab13c6423956de0edf62e"}
Sep 10 03:02:33.230441 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:33.230388    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449" podStartSLOduration=2.230372937 podStartE2EDuration="2.230372937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:02:33.230043846 +0000 UTC m=+950.379943544" watchObservedRunningTime="2026-09-10 03:02:33.230372937 +0000 UTC m=+950.380272633"
Sep 10 03:02:42.025200 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:42.025143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:42.025200 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:42.025209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:42.037529 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:42.037504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:42.273979 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:42.273953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:02:47.122310 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:02:47.122273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:03:03.415695 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:03.415664    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:03:32.110957 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.110926    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"]
Sep 10 03:03:32.114559 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.114539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.117840 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.117811    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-af329f\"/\"default-dockercfg-8lmw2\""
Sep 10 03:03:32.119189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.119154    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-af329f\"/\"explicit-model-af329f-kserve-self-signed-certs\""
Sep 10 03:03:32.119189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.119182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-af329f\"/\"kube-root-ca.crt\""
Sep 10 03:03:32.119374 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.119182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-af329f\"/\"openshift-service-ca.crt\""
Sep 10 03:03:32.127387 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.127362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"]
Sep 10 03:03:32.176692 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.176660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-home\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.176846 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.176705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tls-certs\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.176846 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.176767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-model-cache\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.176846 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.176827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957cp\" (UniqueName: \"kubernetes.io/projected/46048c1d-b8ce-4c43-aac5-4bbdbe684946-kube-api-access-957cp\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.176846 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.176845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-dshm\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.177000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.176883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tmp-dir\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278430 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-home\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278605 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tls-certs\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278605 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-model-cache\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278605 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957cp\" (UniqueName: \"kubernetes.io/projected/46048c1d-b8ce-4c43-aac5-4bbdbe684946-kube-api-access-957cp\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278605 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-dshm\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278883 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tmp-dir\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278883 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-home\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278987 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-model-cache\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.278987 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.278965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tmp-dir\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.280813 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.280792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-dshm\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.281093 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.281076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tls-certs\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.286593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.286571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-957cp\" (UniqueName: \"kubernetes.io/projected/46048c1d-b8ce-4c43-aac5-4bbdbe684946-kube-api-access-957cp\") pod \"explicit-model-af329f-kserve-674bb8ddb7-l52mc\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") " pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.425047 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.424965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:32.697531 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:32.697507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"]
Sep 10 03:03:32.699507 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:03:32.699482    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46048c1d_b8ce_4c43_aac5_4bbdbe684946.slice/crio-dffb8f06a91a0ffa08fb5cd8ad7d3af312f6981761ad810c98abb3e641c3b355 WatchSource:0}: Error finding container dffb8f06a91a0ffa08fb5cd8ad7d3af312f6981761ad810c98abb3e641c3b355: Status 404 returned error can't find the container with id dffb8f06a91a0ffa08fb5cd8ad7d3af312f6981761ad810c98abb3e641c3b355
Sep 10 03:03:33.549613 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:33.549576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc" event={"ID":"46048c1d-b8ce-4c43-aac5-4bbdbe684946","Type":"ContainerStarted","Data":"f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585"}
Sep 10 03:03:33.549613 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:33.549622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc" event={"ID":"46048c1d-b8ce-4c43-aac5-4bbdbe684946","Type":"ContainerStarted","Data":"dffb8f06a91a0ffa08fb5cd8ad7d3af312f6981761ad810c98abb3e641c3b355"}
Sep 10 03:03:33.567704 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:33.567653    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc" podStartSLOduration=1.567638085 podStartE2EDuration="1.567638085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:03:33.565848105 +0000 UTC m=+1010.715747802" watchObservedRunningTime="2026-09-10 03:03:33.567638085 +0000 UTC m=+1010.717537800"
Sep 10 03:03:34.116325 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:34.116288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:03:40.717039 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.717006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"]
Sep 10 03:03:40.720996 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.720979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.723925 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.723904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"distinct-w0-097e16-kserve-self-signed-certs\""
Sep 10 03:03:40.732719 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.732696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"]
Sep 10 03:03:40.860963 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.860928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tls-certs\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.861143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.860989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-home\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.861143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.861020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-dshm\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.861143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.861048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tmp-dir\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.861143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.861069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-model-cache\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.861322 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.861194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrng\" (UniqueName: \"kubernetes.io/projected/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-kube-api-access-pjrng\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.961754 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.961711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-home\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.961754 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.961756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-dshm\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.962009 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.961806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tmp-dir\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.962009 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.961839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-model-cache\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.962009 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.961922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrng\" (UniqueName: \"kubernetes.io/projected/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-kube-api-access-pjrng\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.962009 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.961981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tls-certs\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.962272 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.962195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-home\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.962272 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.962213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tmp-dir\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.962389 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.962291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-model-cache\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.963966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.963945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-dshm\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.964373 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.964356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tls-certs\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:40.970864 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:40.970815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrng\" (UniqueName: \"kubernetes.io/projected/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-kube-api-access-pjrng\") pod \"distinct-w0-097e16-kserve-57ff44599-vpwrw\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:41.031605 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:41.031571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:41.302530 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:41.302501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"]
Sep 10 03:03:41.304655 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:03:41.304628    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bea1e32_b5c9_4f76_8bc2_894f24cf3a37.slice/crio-481a963a8cbea3817852c4baadf967684611eece54f60d29abf78316c3b16cbc WatchSource:0}: Error finding container 481a963a8cbea3817852c4baadf967684611eece54f60d29abf78316c3b16cbc: Status 404 returned error can't find the container with id 481a963a8cbea3817852c4baadf967684611eece54f60d29abf78316c3b16cbc
Sep 10 03:03:41.597107 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:41.597070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw" event={"ID":"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37","Type":"ContainerStarted","Data":"fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598"}
Sep 10 03:03:41.597333 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:41.597117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw" event={"ID":"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37","Type":"ContainerStarted","Data":"481a963a8cbea3817852c4baadf967684611eece54f60d29abf78316c3b16cbc"}
Sep 10 03:03:41.616955 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:41.616904    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw" podStartSLOduration=1.616889584 podStartE2EDuration="1.616889584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:03:41.615108983 +0000 UTC m=+1018.765008680" watchObservedRunningTime="2026-09-10 03:03:41.616889584 +0000 UTC m=+1018.766789290"
Sep 10 03:03:42.425787 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:42.425754    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:42.426237 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:42.425803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:42.438399 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:42.438375    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:42.614133 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:42.614104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:03:51.032421 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:51.032376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:51.032835 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:51.032440    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:51.044917 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:51.044888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:51.665891 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:51.665860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:03:55.420941 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:03:55.420912    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:04:15.877646 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:15.877612    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"]
Sep 10 03:04:15.878107 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:15.877844    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99" podUID="b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" containerName="payload-processing" containerID="cri-o://dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f" gracePeriod=30
Sep 10 03:04:15.885632 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:15.885495    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"]
Sep 10 03:04:15.886131 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:15.886070    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w" podUID="7abecd37-49ec-4f63-b2d6-07640f536cec" containerName="payload-pre-processing" containerID="cri-o://7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d" gracePeriod=30
Sep 10 03:04:16.171718 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.171693    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:04:16.174786 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.174767    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:04:16.321427 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.321392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-plugins-config-volume\") pod \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") "
Sep 10 03:04:16.321608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.321439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g6gq\" (UniqueName: \"kubernetes.io/projected/7abecd37-49ec-4f63-b2d6-07640f536cec-kube-api-access-5g6gq\") pod \"7abecd37-49ec-4f63-b2d6-07640f536cec\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") "
Sep 10 03:04:16.321608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.321484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mhxh\" (UniqueName: \"kubernetes.io/projected/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-kube-api-access-4mhxh\") pod \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\" (UID: \"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67\") "
Sep 10 03:04:16.321608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.321514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abecd37-49ec-4f63-b2d6-07640f536cec-plugins-config-volume\") pod \"7abecd37-49ec-4f63-b2d6-07640f536cec\" (UID: \"7abecd37-49ec-4f63-b2d6-07640f536cec\") "
Sep 10 03:04:16.321843 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.321818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" (UID: "b1c5f35d-7b77-49f5-9eaa-df85cdb9af67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:04:16.321911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.321892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abecd37-49ec-4f63-b2d6-07640f536cec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7abecd37-49ec-4f63-b2d6-07640f536cec" (UID: "7abecd37-49ec-4f63-b2d6-07640f536cec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:04:16.323567 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.323546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abecd37-49ec-4f63-b2d6-07640f536cec-kube-api-access-5g6gq" (OuterVolumeSpecName: "kube-api-access-5g6gq") pod "7abecd37-49ec-4f63-b2d6-07640f536cec" (UID: "7abecd37-49ec-4f63-b2d6-07640f536cec"). InnerVolumeSpecName "kube-api-access-5g6gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:04:16.323669 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.323629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-kube-api-access-4mhxh" (OuterVolumeSpecName: "kube-api-access-4mhxh") pod "b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" (UID: "b1c5f35d-7b77-49f5-9eaa-df85cdb9af67"). InnerVolumeSpecName "kube-api-access-4mhxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:04:16.423152 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.423063    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:16.423152 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.423096    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g6gq\" (UniqueName: \"kubernetes.io/projected/7abecd37-49ec-4f63-b2d6-07640f536cec-kube-api-access-5g6gq\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:16.423152 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.423110    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhxh\" (UniqueName: \"kubernetes.io/projected/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67-kube-api-access-4mhxh\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:16.423152 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.423124    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abecd37-49ec-4f63-b2d6-07640f536cec-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:16.794135 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.794042    2580 generic.go:356] "Generic (PLEG): container finished" podID="7abecd37-49ec-4f63-b2d6-07640f536cec" containerID="7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d" exitCode=0
Sep 10 03:04:16.794135 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.794085    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"
Sep 10 03:04:16.794462 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.794128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w" event={"ID":"7abecd37-49ec-4f63-b2d6-07640f536cec","Type":"ContainerDied","Data":"7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d"}
Sep 10 03:04:16.794462 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.794203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w" event={"ID":"7abecd37-49ec-4f63-b2d6-07640f536cec","Type":"ContainerDied","Data":"77e6d9e578142ec03a44bc8d11674ac0d73c9eba8567fc7899f2d1ad0088d0f6"}
Sep 10 03:04:16.794462 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.794232    2580 scope.go:117] "RemoveContainer" containerID="7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d"
Sep 10 03:04:16.795644 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.795613    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" containerID="dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f" exitCode=0
Sep 10 03:04:16.795770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.795653    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"
Sep 10 03:04:16.795770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.795712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99" event={"ID":"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67","Type":"ContainerDied","Data":"dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f"}
Sep 10 03:04:16.795770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.795754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99" event={"ID":"b1c5f35d-7b77-49f5-9eaa-df85cdb9af67","Type":"ContainerDied","Data":"ee22ea4491291677e316abb876685f5c2ceee8178b7e2139d414745528a54c23"}
Sep 10 03:04:16.805120 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.805103    2580 scope.go:117] "RemoveContainer" containerID="7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d"
Sep 10 03:04:16.805499 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:04:16.805395    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d\": container with ID starting with 7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d not found: ID does not exist" containerID="7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d"
Sep 10 03:04:16.805499 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.805419    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d"} err="failed to get container status \"7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d\": rpc error: code = NotFound desc = could not find container \"7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d\": container with ID starting with 7dc66933cb5b941817f10752f9b3b90ddaf14813f74c23bb8639b441d9418a5d not found: ID does not exist"
Sep 10 03:04:16.805499 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.805436    2580 scope.go:117] "RemoveContainer" containerID="dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f"
Sep 10 03:04:16.813778 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.813740    2580 scope.go:117] "RemoveContainer" containerID="dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f"
Sep 10 03:04:16.814180 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:04:16.814144    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f\": container with ID starting with dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f not found: ID does not exist" containerID="dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f"
Sep 10 03:04:16.814252 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.814190    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f"} err="failed to get container status \"dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f\": rpc error: code = NotFound desc = could not find container \"dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f\": container with ID starting with dc36e5505c4e34e818ce8a0a2cedf3de6ad141d9a2de7ff193496fcad095633f not found: ID does not exist"
Sep 10 03:04:16.816346 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.816328    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"]
Sep 10 03:04:16.819889 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.819868    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-af329f-66bfc59fc8-5tg4w"]
Sep 10 03:04:16.830652 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.830615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"]
Sep 10 03:04:16.839088 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:16.839060    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-af329f-6c6c696665-f6x99"]
Sep 10 03:04:17.325536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:17.325504    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abecd37-49ec-4f63-b2d6-07640f536cec" path="/var/lib/kubelet/pods/7abecd37-49ec-4f63-b2d6-07640f536cec/volumes"
Sep 10 03:04:17.325909 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:17.325874    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" path="/var/lib/kubelet/pods/b1c5f35d-7b77-49f5-9eaa-df85cdb9af67/volumes"
Sep 10 03:04:20.601393 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.601361    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"]
Sep 10 03:04:20.601788 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.601775    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" containerName="payload-processing"
Sep 10 03:04:20.601830 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.601790    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" containerName="payload-processing"
Sep 10 03:04:20.601830 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.601800    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abecd37-49ec-4f63-b2d6-07640f536cec" containerName="payload-pre-processing"
Sep 10 03:04:20.601830 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.601805    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abecd37-49ec-4f63-b2d6-07640f536cec" containerName="payload-pre-processing"
Sep 10 03:04:20.601926 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.601883    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abecd37-49ec-4f63-b2d6-07640f536cec" containerName="payload-pre-processing"
Sep 10 03:04:20.601926 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.601891    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c5f35d-7b77-49f5-9eaa-df85cdb9af67" containerName="payload-processing"
Sep 10 03:04:20.606439 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.606422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.609368 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.609347    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"distinct-w4-e6e6b8-kserve-self-signed-certs\""
Sep 10 03:04:20.613769 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.613748    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"]
Sep 10 03:04:20.767636 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.767539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cslnt\" (UniqueName: \"kubernetes.io/projected/ee2514ee-183d-49f4-9d34-d7bcbab49933-kube-api-access-cslnt\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.767786 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.767659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-tmp-dir\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.767786 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.767687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-dshm\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.767786 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.767720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2514ee-183d-49f4-9d34-d7bcbab49933-tls-certs\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.767786 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.767755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-model-cache\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.767919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.767862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-home\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.868986 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.868952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-tmp-dir\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.868986 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.868989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-dshm\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.869292 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.869030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2514ee-183d-49f4-9d34-d7bcbab49933-tls-certs\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.869292 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.869073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-model-cache\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.869292 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.869131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-home\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.869292 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.869159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cslnt\" (UniqueName: \"kubernetes.io/projected/ee2514ee-183d-49f4-9d34-d7bcbab49933-kube-api-access-cslnt\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.870085 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.869919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-tmp-dir\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.870085 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.869996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-home\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.870569 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.870544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-model-cache\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.873032 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.872875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2514ee-183d-49f4-9d34-d7bcbab49933-tls-certs\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.873153 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.873091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-dshm\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.877711 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.877686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cslnt\" (UniqueName: \"kubernetes.io/projected/ee2514ee-183d-49f4-9d34-d7bcbab49933-kube-api-access-cslnt\") pod \"distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:20.917454 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:20.917421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:21.190916 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:21.190889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"]
Sep 10 03:04:21.192444 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:04:21.192419    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee2514ee_183d_49f4_9d34_d7bcbab49933.slice/crio-be0e2e8d8a125f217dc58027985b8ae0d003b8da776552c79bfed2ca28ff9378 WatchSource:0}: Error finding container be0e2e8d8a125f217dc58027985b8ae0d003b8da776552c79bfed2ca28ff9378: Status 404 returned error can't find the container with id be0e2e8d8a125f217dc58027985b8ae0d003b8da776552c79bfed2ca28ff9378
Sep 10 03:04:21.476561 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:21.476474    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"]
Sep 10 03:04:21.827816 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:21.827722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q" event={"ID":"ee2514ee-183d-49f4-9d34-d7bcbab49933","Type":"ContainerStarted","Data":"ed552828165af892f79ad119c1d989cf23bb6df6f9b770ab58582da96377bf09"}
Sep 10 03:04:21.827816 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:21.827773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q" event={"ID":"ee2514ee-183d-49f4-9d34-d7bcbab49933","Type":"ContainerStarted","Data":"be0e2e8d8a125f217dc58027985b8ae0d003b8da776552c79bfed2ca28ff9378"}
Sep 10 03:04:21.843948 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:21.843901    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q" podStartSLOduration=1.843885415 podStartE2EDuration="1.843885415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:21.843445171 +0000 UTC m=+1058.993344859" watchObservedRunningTime="2026-09-10 03:04:21.843885415 +0000 UTC m=+1058.993785112"
Sep 10 03:04:24.117180 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:24.117132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:04:30.917924 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:30.917892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:30.918350 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:30.917939    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:30.930114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:30.930084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:31.896006 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:31.895978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:04:36.496816 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.496711    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc" podUID="46048c1d-b8ce-4c43-aac5-4bbdbe684946" containerName="main" containerID="cri-o://f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585" gracePeriod=45
Sep 10 03:04:36.747156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.747085    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:04:36.826449 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-home\") pod \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") "
Sep 10 03:04:36.826449 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826453    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-model-cache\") pod \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") "
Sep 10 03:04:36.826726 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-957cp\" (UniqueName: \"kubernetes.io/projected/46048c1d-b8ce-4c43-aac5-4bbdbe684946-kube-api-access-957cp\") pod \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") "
Sep 10 03:04:36.826726 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-dshm\") pod \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") "
Sep 10 03:04:36.826726 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tmp-dir\") pod \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") "
Sep 10 03:04:36.826726 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tls-certs\") pod \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\" (UID: \"46048c1d-b8ce-4c43-aac5-4bbdbe684946\") "
Sep 10 03:04:36.826930 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-model-cache" (OuterVolumeSpecName: "model-cache") pod "46048c1d-b8ce-4c43-aac5-4bbdbe684946" (UID: "46048c1d-b8ce-4c43-aac5-4bbdbe684946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:04:36.826930 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-home" (OuterVolumeSpecName: "home") pod "46048c1d-b8ce-4c43-aac5-4bbdbe684946" (UID: "46048c1d-b8ce-4c43-aac5-4bbdbe684946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:04:36.826930 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46048c1d-b8ce-4c43-aac5-4bbdbe684946" (UID: "46048c1d-b8ce-4c43-aac5-4bbdbe684946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:04:36.827083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826975    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:36.827083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826986    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:36.827083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.826996    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:36.828785 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.828755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-dshm" (OuterVolumeSpecName: "dshm") pod "46048c1d-b8ce-4c43-aac5-4bbdbe684946" (UID: "46048c1d-b8ce-4c43-aac5-4bbdbe684946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:04:36.828905 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.828801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46048c1d-b8ce-4c43-aac5-4bbdbe684946" (UID: "46048c1d-b8ce-4c43-aac5-4bbdbe684946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:04:36.828905 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.828841    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46048c1d-b8ce-4c43-aac5-4bbdbe684946-kube-api-access-957cp" (OuterVolumeSpecName: "kube-api-access-957cp") pod "46048c1d-b8ce-4c43-aac5-4bbdbe684946" (UID: "46048c1d-b8ce-4c43-aac5-4bbdbe684946"). InnerVolumeSpecName "kube-api-access-957cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:04:36.913667 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.913628    2580 generic.go:356] "Generic (PLEG): container finished" podID="46048c1d-b8ce-4c43-aac5-4bbdbe684946" containerID="f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585" exitCode=0
Sep 10 03:04:36.913836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.913684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"
Sep 10 03:04:36.913836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.913708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc" event={"ID":"46048c1d-b8ce-4c43-aac5-4bbdbe684946","Type":"ContainerDied","Data":"f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585"}
Sep 10 03:04:36.913836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.913752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc" event={"ID":"46048c1d-b8ce-4c43-aac5-4bbdbe684946","Type":"ContainerDied","Data":"dffb8f06a91a0ffa08fb5cd8ad7d3af312f6981761ad810c98abb3e641c3b355"}
Sep 10 03:04:36.913836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.913769    2580 scope.go:117] "RemoveContainer" containerID="f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585"
Sep 10 03:04:36.924055 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.924032    2580 scope.go:117] "RemoveContainer" containerID="f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585"
Sep 10 03:04:36.924439 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:04:36.924414    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585\": container with ID starting with f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585 not found: ID does not exist" containerID="f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585"
Sep 10 03:04:36.924561 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.924449    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585"} err="failed to get container status \"f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585\": rpc error: code = NotFound desc = could not find container \"f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585\": container with ID starting with f4b5b97e5ccaf52cb81cfca3f08d9098e25d09bb4e0e17d5e8e7a01adf6d8585 not found: ID does not exist"
Sep 10 03:04:36.928450 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.928422    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957cp\" (UniqueName: \"kubernetes.io/projected/46048c1d-b8ce-4c43-aac5-4bbdbe684946-kube-api-access-957cp\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:36.928450 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.928448    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46048c1d-b8ce-4c43-aac5-4bbdbe684946-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:36.928612 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.928457    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46048c1d-b8ce-4c43-aac5-4bbdbe684946-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:04:36.935223 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.935192    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"]
Sep 10 03:04:36.938996 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:36.938972    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-af329f/explicit-model-af329f-kserve-674bb8ddb7-l52mc"]
Sep 10 03:04:37.325968 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:37.325935    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46048c1d-b8ce-4c43-aac5-4bbdbe684946" path="/var/lib/kubelet/pods/46048c1d-b8ce-4c43-aac5-4bbdbe684946/volumes"
Sep 10 03:04:50.919112 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:50.919077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:04:55.481176 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.481130    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"]
Sep 10 03:04:55.481646 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.481609    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46048c1d-b8ce-4c43-aac5-4bbdbe684946" containerName="main"
Sep 10 03:04:55.481646 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.481622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46048c1d-b8ce-4c43-aac5-4bbdbe684946" containerName="main"
Sep 10 03:04:55.481759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.481700    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="46048c1d-b8ce-4c43-aac5-4bbdbe684946" containerName="main"
Sep 10 03:04:55.486419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.486402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.489460 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.489436    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c3058d-dockercfg-gs4wq\""
Sep 10 03:04:55.489460 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.489453    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c3058d\""
Sep 10 03:04:55.495945 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.495919    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"]
Sep 10 03:04:55.526114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.526083    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"]
Sep 10 03:04:55.529619 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.529604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:55.539243 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.539219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"]
Sep 10 03:04:55.623787 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.623749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/210bd681-22ec-44fb-aef3-ac977f5c9fc0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:55.624000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.623912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czsl\" (UniqueName: \"kubernetes.io/projected/31c2467b-86d5-4c28-ab85-44838bddef71-kube-api-access-8czsl\") pod \"payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.624000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.623960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c2467b-86d5-4c28-ab85-44838bddef71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.624105 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.624018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg229\" (UniqueName: \"kubernetes.io/projected/210bd681-22ec-44fb-aef3-ac977f5c9fc0-kube-api-access-sg229\") pod \"payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:55.724970 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.724920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czsl\" (UniqueName: \"kubernetes.io/projected/31c2467b-86d5-4c28-ab85-44838bddef71-kube-api-access-8czsl\") pod \"payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.724970 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.724983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c2467b-86d5-4c28-ab85-44838bddef71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.725276 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.725005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg229\" (UniqueName: \"kubernetes.io/projected/210bd681-22ec-44fb-aef3-ac977f5c9fc0-kube-api-access-sg229\") pod \"payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:55.725276 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.725044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/210bd681-22ec-44fb-aef3-ac977f5c9fc0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:55.725740 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.725717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/210bd681-22ec-44fb-aef3-ac977f5c9fc0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:55.725833 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.725789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c2467b-86d5-4c28-ab85-44838bddef71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.733459 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.733402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czsl\" (UniqueName: \"kubernetes.io/projected/31c2467b-86d5-4c28-ab85-44838bddef71-kube-api-access-8czsl\") pod \"payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.733538 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.733470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg229\" (UniqueName: \"kubernetes.io/projected/210bd681-22ec-44fb-aef3-ac977f5c9fc0-kube-api-access-sg229\") pod \"payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:55.797377 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.797344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:55.840747 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:55.840715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:56.065710 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:56.065659    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"]
Sep 10 03:04:56.067971 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:04:56.067947    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c2467b_86d5_4c28_ab85_44838bddef71.slice/crio-64e2cf5621faf93e00cfaa151026928cf58f5f0bd92e3352b17ed6f08d74884e WatchSource:0}: Error finding container 64e2cf5621faf93e00cfaa151026928cf58f5f0bd92e3352b17ed6f08d74884e: Status 404 returned error can't find the container with id 64e2cf5621faf93e00cfaa151026928cf58f5f0bd92e3352b17ed6f08d74884e
Sep 10 03:04:56.121417 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:56.121328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"]
Sep 10 03:04:56.125700 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:04:56.125676    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210bd681_22ec_44fb_aef3_ac977f5c9fc0.slice/crio-f593f39177e4a1e2e18661627d704bbe2a2e1b76cdbb6a1b324a6289af3e9356 WatchSource:0}: Error finding container f593f39177e4a1e2e18661627d704bbe2a2e1b76cdbb6a1b324a6289af3e9356: Status 404 returned error can't find the container with id f593f39177e4a1e2e18661627d704bbe2a2e1b76cdbb6a1b324a6289af3e9356
Sep 10 03:04:57.023313 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.023274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs" event={"ID":"31c2467b-86d5-4c28-ab85-44838bddef71","Type":"ContainerStarted","Data":"c094912d5bf9604bf307235830de4e951d3d52ed43688ab60ac3419199290830"}
Sep 10 03:04:57.023762 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.023341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs" event={"ID":"31c2467b-86d5-4c28-ab85-44838bddef71","Type":"ContainerStarted","Data":"64e2cf5621faf93e00cfaa151026928cf58f5f0bd92e3352b17ed6f08d74884e"}
Sep 10 03:04:57.023762 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.023372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:04:57.024967 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.024942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592" event={"ID":"210bd681-22ec-44fb-aef3-ac977f5c9fc0","Type":"ContainerStarted","Data":"7dfd91109781b091b8c16e8bf6d63789253afc27e6bfa16e3152d5ea3161f0e6"}
Sep 10 03:04:57.025085 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.024981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592" event={"ID":"210bd681-22ec-44fb-aef3-ac977f5c9fc0","Type":"ContainerStarted","Data":"f593f39177e4a1e2e18661627d704bbe2a2e1b76cdbb6a1b324a6289af3e9356"}
Sep 10 03:04:57.025085 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.025051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:04:57.040263 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.040217    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs" podStartSLOduration=2.040203206 podStartE2EDuration="2.040203206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:57.038778254 +0000 UTC m=+1094.188677954" watchObservedRunningTime="2026-09-10 03:04:57.040203206 +0000 UTC m=+1094.190102903"
Sep 10 03:04:57.055449 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:57.055402    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592" podStartSLOduration=2.055391332 podStartE2EDuration="2.055391332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:57.053379676 +0000 UTC m=+1094.203279372" watchObservedRunningTime="2026-09-10 03:04:57.055391332 +0000 UTC m=+1094.205291056"
Sep 10 03:04:59.262437 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.262404    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"]
Sep 10 03:04:59.266952 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.266930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.269832 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.269807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-9b8682-openshift-default-dockercfg-rqlcm\""
Sep 10 03:04:59.278126 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.278095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"]
Sep 10 03:04:59.366645 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-istio-token\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.366850 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-envoy\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.366850 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-certs\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.366850 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-socket\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.366850 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98d2139-403e-4d58-89f9-f5e95640c379-istio-podinfo\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.366850 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-data\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.367118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98d2139-403e-4d58-89f9-f5e95640c379-istiod-ca-cert\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.367118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-credential-socket\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.367118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.366954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9dll\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-kube-api-access-r9dll\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468679 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98d2139-403e-4d58-89f9-f5e95640c379-istiod-ca-cert\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468679 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-credential-socket\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468679 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9dll\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-kube-api-access-r9dll\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-istio-token\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-envoy\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-certs\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-socket\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98d2139-403e-4d58-89f9-f5e95640c379-istio-podinfo\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.468989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.468930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-data\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.469444 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.469395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-certs\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.469570 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.469456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-credential-socket\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.469570 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.469479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98d2139-403e-4d58-89f9-f5e95640c379-istiod-ca-cert\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.469570 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.469489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-socket\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.469570 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.469502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-data\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.471259 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.471231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-envoy\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.471409 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.471393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98d2139-403e-4d58-89f9-f5e95640c379-istio-podinfo\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.478286 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.478260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9dll\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-kube-api-access-r9dll\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.478436 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.478417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-istio-token\") pod \"e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") " pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.579860 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.579733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:04:59.858972 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.858945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"]
Sep 10 03:04:59.860930 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:04:59.860900    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98d2139_403e_4d58_89f9_f5e95640c379.slice/crio-b021b465ea09caff71e62ae95c23494a7b90e9edbb5625db8cf14a77a3b2c0fd WatchSource:0}: Error finding container b021b465ea09caff71e62ae95c23494a7b90e9edbb5625db8cf14a77a3b2c0fd: Status 404 returned error can't find the container with id b021b465ea09caff71e62ae95c23494a7b90e9edbb5625db8cf14a77a3b2c0fd
Sep 10 03:04:59.863284 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.863253    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:04:59.863390 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.863314    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:04:59.863390 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:04:59.863343    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:05:00.053607 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:00.053572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n" event={"ID":"c98d2139-403e-4d58-89f9-f5e95640c379","Type":"ContainerStarted","Data":"dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56"}
Sep 10 03:05:00.053790 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:00.053628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n" event={"ID":"c98d2139-403e-4d58-89f9-f5e95640c379","Type":"ContainerStarted","Data":"b021b465ea09caff71e62ae95c23494a7b90e9edbb5625db8cf14a77a3b2c0fd"}
Sep 10 03:05:00.082362 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:00.082303    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n" podStartSLOduration=1.082287405 podStartE2EDuration="1.082287405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:00.079623576 +0000 UTC m=+1097.229523284" watchObservedRunningTime="2026-09-10 03:05:00.082287405 +0000 UTC m=+1097.232187102"
Sep 10 03:05:00.580506 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:00.580466    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:05:00.586451 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:00.586420    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:05:01.059225 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:01.059194    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:05:01.060358 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:01.060335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:05:08.032961 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:08.032927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:05:08.033349 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:08.032999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:05:18.781500 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:18.781468    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8012-1f6d92"]
Sep 10 03:05:18.785139 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:18.785115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:18.787875 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:18.787853    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mtn2b\""
Sep 10 03:05:18.789035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:18.789013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-1f6d92"]
Sep 10 03:05:18.873667 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:18.873612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zbp\" (UniqueName: \"kubernetes.io/projected/1bbe4bdb-904b-4f16-a74b-af40c28d4f5b-kube-api-access-77zbp\") pod \"test-curl-8012-1f6d92\" (UID: \"1bbe4bdb-904b-4f16-a74b-af40c28d4f5b\") " pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:18.974934 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:18.974900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zbp\" (UniqueName: \"kubernetes.io/projected/1bbe4bdb-904b-4f16-a74b-af40c28d4f5b-kube-api-access-77zbp\") pod \"test-curl-8012-1f6d92\" (UID: \"1bbe4bdb-904b-4f16-a74b-af40c28d4f5b\") " pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:18.983713 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:18.983686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zbp\" (UniqueName: \"kubernetes.io/projected/1bbe4bdb-904b-4f16-a74b-af40c28d4f5b-kube-api-access-77zbp\") pod \"test-curl-8012-1f6d92\" (UID: \"1bbe4bdb-904b-4f16-a74b-af40c28d4f5b\") " pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:19.097045 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:19.097012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:19.366444 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:19.366422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-1f6d92"]
Sep 10 03:05:19.367976 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:05:19.367940    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bbe4bdb_904b_4f16_a74b_af40c28d4f5b.slice/crio-5be759187d92fbca0732e0bd5daa72b2856322dd0dbe3f85b0644f48c12669e7 WatchSource:0}: Error finding container 5be759187d92fbca0732e0bd5daa72b2856322dd0dbe3f85b0644f48c12669e7: Status 404 returned error can't find the container with id 5be759187d92fbca0732e0bd5daa72b2856322dd0dbe3f85b0644f48c12669e7
Sep 10 03:05:20.181992 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.181948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-1f6d92" event={"ID":"1bbe4bdb-904b-4f16-a74b-af40c28d4f5b","Type":"ContainerStarted","Data":"a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce"}
Sep 10 03:05:20.182457 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.182002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-1f6d92" event={"ID":"1bbe4bdb-904b-4f16-a74b-af40c28d4f5b","Type":"ContainerStarted","Data":"5be759187d92fbca0732e0bd5daa72b2856322dd0dbe3f85b0644f48c12669e7"}
Sep 10 03:05:20.195939 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.195897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8012-1f6d92" podStartSLOduration=2.195883899 podStartE2EDuration="2.195883899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:20.195038995 +0000 UTC m=+1117.344938692" watchObservedRunningTime="2026-09-10 03:05:20.195883899 +0000 UTC m=+1117.345783595"
Sep 10 03:05:20.863446 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.863413    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8012-1f6d92"]
Sep 10 03:05:20.867482 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.867448    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8012-1f6d92"]
Sep 10 03:05:20.987568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.987536    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8012-614328"]
Sep 10 03:05:20.988042 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.988026    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" containerName="test-curl-8012-1f6d92"
Sep 10 03:05:20.988084 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.988043    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" containerName="test-curl-8012-1f6d92"
Sep 10 03:05:20.988132 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.988123    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" containerName="test-curl-8012-1f6d92"
Sep 10 03:05:20.991186 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.991157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:20.997036 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:20.997011    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-614328"]
Sep 10 03:05:21.096663 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:21.096632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v899d\" (UniqueName: \"kubernetes.io/projected/a0d325ee-e746-4e2c-a064-f8bc8a107a4d-kube-api-access-v899d\") pod \"test-curl-8012-614328\" (UID: \"a0d325ee-e746-4e2c-a064-f8bc8a107a4d\") " pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:21.188224 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:05:21.188128    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8012-1f6d92" container="test-curl-8012-1f6d92"
Sep 10 03:05:21.197266 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:21.197243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v899d\" (UniqueName: \"kubernetes.io/projected/a0d325ee-e746-4e2c-a064-f8bc8a107a4d-kube-api-access-v899d\") pod \"test-curl-8012-614328\" (UID: \"a0d325ee-e746-4e2c-a064-f8bc8a107a4d\") " pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:21.207052 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:21.207030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v899d\" (UniqueName: \"kubernetes.io/projected/a0d325ee-e746-4e2c-a064-f8bc8a107a4d-kube-api-access-v899d\") pod \"test-curl-8012-614328\" (UID: \"a0d325ee-e746-4e2c-a064-f8bc8a107a4d\") " pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:21.301841 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:21.301811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:21.506341 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:21.506257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:05:21.570615 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:21.570591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-614328"]
Sep 10 03:05:21.572320 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:05:21.572291    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d325ee_e746_4e2c_a064_f8bc8a107a4d.slice/crio-0ce2a72223cb7c7b14702e851ed9b271ad2df501077796118d6b404ee411d1a3 WatchSource:0}: Error finding container 0ce2a72223cb7c7b14702e851ed9b271ad2df501077796118d6b404ee411d1a3: Status 404 returned error can't find the container with id 0ce2a72223cb7c7b14702e851ed9b271ad2df501077796118d6b404ee411d1a3
Sep 10 03:05:22.195746 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:22.195708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-614328" event={"ID":"a0d325ee-e746-4e2c-a064-f8bc8a107a4d","Type":"ContainerStarted","Data":"137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4"}
Sep 10 03:05:22.196237 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:22.195758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-614328" event={"ID":"a0d325ee-e746-4e2c-a064-f8bc8a107a4d","Type":"ContainerStarted","Data":"0ce2a72223cb7c7b14702e851ed9b271ad2df501077796118d6b404ee411d1a3"}
Sep 10 03:05:22.196237 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:22.195893    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8012-1f6d92" podUID="1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" containerName="test-curl-8012-1f6d92" containerID="cri-o://a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce" gracePeriod=2
Sep 10 03:05:22.209039 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:22.208990    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8012-614328" podStartSLOduration=2.208973215 podStartE2EDuration="2.208973215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:22.208730802 +0000 UTC m=+1119.358630501" watchObservedRunningTime="2026-09-10 03:05:22.208973215 +0000 UTC m=+1119.358872912"
Sep 10 03:05:22.823882 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:22.823847    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8012-614328"]
Sep 10 03:05:22.827136 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:22.827108    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8012-614328"]
Sep 10 03:05:23.201918 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:05:23.201886    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8012-614328" container="test-curl-8012-614328"
Sep 10 03:05:24.207956 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:24.207915    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8012-614328" podUID="a0d325ee-e746-4e2c-a064-f8bc8a107a4d" containerName="test-curl-8012-614328" containerID="cri-o://137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4" gracePeriod=2
Sep 10 03:05:24.386933 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:24.386908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:24.533458 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:24.533375    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77zbp\" (UniqueName: \"kubernetes.io/projected/1bbe4bdb-904b-4f16-a74b-af40c28d4f5b-kube-api-access-77zbp\") pod \"1bbe4bdb-904b-4f16-a74b-af40c28d4f5b\" (UID: \"1bbe4bdb-904b-4f16-a74b-af40c28d4f5b\") "
Sep 10 03:05:24.535457 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:24.535435    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbe4bdb-904b-4f16-a74b-af40c28d4f5b-kube-api-access-77zbp" (OuterVolumeSpecName: "kube-api-access-77zbp") pod "1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" (UID: "1bbe4bdb-904b-4f16-a74b-af40c28d4f5b"). InnerVolumeSpecName "kube-api-access-77zbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:05:24.635148 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:24.635109    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zbp\" (UniqueName: \"kubernetes.io/projected/1bbe4bdb-904b-4f16-a74b-af40c28d4f5b-kube-api-access-77zbp\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:25.215102 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:25.215069    2580 generic.go:356] "Generic (PLEG): container finished" podID="1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" containerID="a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce" exitCode=137
Sep 10 03:05:25.215102 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:25.215104    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:25.215571 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:25.215159    2580 scope.go:117] "RemoveContainer" containerID="a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce"
Sep 10 03:05:25.224396 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:25.224378    2580 scope.go:117] "RemoveContainer" containerID="a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce"
Sep 10 03:05:25.224703 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:05:25.224685    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce\": container with ID starting with a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce not found: ID does not exist" containerID="a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce"
Sep 10 03:05:25.224774 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:25.224716    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce"} err="failed to get container status \"a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce\": rpc error: code = NotFound desc = could not find container \"a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce\": container with ID starting with a09423cbdaf8299a53cc8e7b6616022510421cb47bd5cc94266f739d70f6f5ce not found: ID does not exist"
Sep 10 03:05:25.225803 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:05:25.225782    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8012-1f6d92\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" pod="opendatahub/test-curl-8012-1f6d92"
Sep 10 03:05:25.325561 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:25.325528    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bbe4bdb-904b-4f16-a74b-af40c28d4f5b" path="/var/lib/kubelet/pods/1bbe4bdb-904b-4f16-a74b-af40c28d4f5b/volumes"
Sep 10 03:05:26.377606 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:26.377583    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:26.453042 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:26.453005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v899d\" (UniqueName: \"kubernetes.io/projected/a0d325ee-e746-4e2c-a064-f8bc8a107a4d-kube-api-access-v899d\") pod \"a0d325ee-e746-4e2c-a064-f8bc8a107a4d\" (UID: \"a0d325ee-e746-4e2c-a064-f8bc8a107a4d\") "
Sep 10 03:05:26.455017 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:26.454987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d325ee-e746-4e2c-a064-f8bc8a107a4d-kube-api-access-v899d" (OuterVolumeSpecName: "kube-api-access-v899d") pod "a0d325ee-e746-4e2c-a064-f8bc8a107a4d" (UID: "a0d325ee-e746-4e2c-a064-f8bc8a107a4d"). InnerVolumeSpecName "kube-api-access-v899d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:05:26.554033 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:26.553994    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v899d\" (UniqueName: \"kubernetes.io/projected/a0d325ee-e746-4e2c-a064-f8bc8a107a4d-kube-api-access-v899d\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:27.229934 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:27.229903    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0d325ee-e746-4e2c-a064-f8bc8a107a4d" containerID="137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4" exitCode=137
Sep 10 03:05:27.230232 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:27.229948    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:27.230232 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:27.230003    2580 scope.go:117] "RemoveContainer" containerID="137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4"
Sep 10 03:05:27.239455 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:27.239440    2580 scope.go:117] "RemoveContainer" containerID="137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4"
Sep 10 03:05:27.239727 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:05:27.239709    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4\": container with ID starting with 137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4 not found: ID does not exist" containerID="137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4"
Sep 10 03:05:27.239791 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:27.239739    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4"} err="failed to get container status \"137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4\": rpc error: code = NotFound desc = could not find container \"137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4\": container with ID starting with 137d09cb56006cb69ebcc01fcc0b928703b0c37183fb59b322bc746760bb58a4 not found: ID does not exist"
Sep 10 03:05:27.240468 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:05:27.240447    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8012-614328\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="a0d325ee-e746-4e2c-a064-f8bc8a107a4d" pod="opendatahub/test-curl-8012-614328"
Sep 10 03:05:27.325511 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:27.325481    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0d325ee-e746-4e2c-a064-f8bc8a107a4d" path="/var/lib/kubelet/pods/a0d325ee-e746-4e2c-a064-f8bc8a107a4d/volumes"
Sep 10 03:05:37.996027 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:37.995991    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"]
Sep 10 03:05:37.996462 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:37.996448    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d325ee-e746-4e2c-a064-f8bc8a107a4d" containerName="test-curl-8012-614328"
Sep 10 03:05:37.996510 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:37.996464    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d325ee-e746-4e2c-a064-f8bc8a107a4d" containerName="test-curl-8012-614328"
Sep 10 03:05:37.996551 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:37.996542    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d325ee-e746-4e2c-a064-f8bc8a107a4d" containerName="test-curl-8012-614328"
Sep 10 03:05:38.001046 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.001026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.005241 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.005214    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"distinct-2-w0-097e16-kserve-self-signed-certs\""
Sep 10 03:05:38.007979 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.007954    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"]
Sep 10 03:05:38.075382 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.075350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-model-cache\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.075382 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.075381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-dshm\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.075608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.075406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k9h\" (UniqueName: \"kubernetes.io/projected/55b16e80-04b5-4743-91e2-5c34e810133a-kube-api-access-n4k9h\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.075608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.075498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-home\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.075608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.075525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-tmp-dir\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.075608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.075563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b16e80-04b5-4743-91e2-5c34e810133a-tls-certs\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177018 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.176978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-model-cache\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-dshm\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k9h\" (UniqueName: \"kubernetes.io/projected/55b16e80-04b5-4743-91e2-5c34e810133a-kube-api-access-n4k9h\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-home\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-tmp-dir\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b16e80-04b5-4743-91e2-5c34e810133a-tls-certs\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-model-cache\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177595 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-home\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.177647 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.177614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-tmp-dir\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.179965 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.179929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b16e80-04b5-4743-91e2-5c34e810133a-tls-certs\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.180090 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.179966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-dshm\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.185087 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.185062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k9h\" (UniqueName: \"kubernetes.io/projected/55b16e80-04b5-4743-91e2-5c34e810133a-kube-api-access-n4k9h\") pod \"distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") " pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.311309 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.311221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:38.584220 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:38.584192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"]
Sep 10 03:05:38.585363 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:05:38.585334    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55b16e80_04b5_4743_91e2_5c34e810133a.slice/crio-7a24457c5d7da37526b580309b65968791969effb0e0d38d365daf91b85bb119 WatchSource:0}: Error finding container 7a24457c5d7da37526b580309b65968791969effb0e0d38d365daf91b85bb119: Status 404 returned error can't find the container with id 7a24457c5d7da37526b580309b65968791969effb0e0d38d365daf91b85bb119
Sep 10 03:05:39.308785 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:39.308741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6" event={"ID":"55b16e80-04b5-4743-91e2-5c34e810133a","Type":"ContainerStarted","Data":"78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c"}
Sep 10 03:05:39.308785 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:39.308785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6" event={"ID":"55b16e80-04b5-4743-91e2-5c34e810133a","Type":"ContainerStarted","Data":"7a24457c5d7da37526b580309b65968791969effb0e0d38d365daf91b85bb119"}
Sep 10 03:05:39.327722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:39.327673    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6" podStartSLOduration=2.327657515 podStartE2EDuration="2.327657515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:39.32666584 +0000 UTC m=+1136.476565537" watchObservedRunningTime="2026-09-10 03:05:39.327657515 +0000 UTC m=+1136.477557213"
Sep 10 03:05:43.393543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:43.393506    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"]
Sep 10 03:05:43.467490 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:43.467459    2580 scope.go:117] "RemoveContainer" containerID="6415172f0ee884f35e60f07a1607ede19ef1037bcc67cd526a92767f185b6fe4"
Sep 10 03:05:46.727412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:46.727376    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:05:48.312101 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:48.312059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:48.312584 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:48.312121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:48.325014 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:48.324981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:48.375025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:48.374995    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:05:58.416026 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.415979    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm" podUID="95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" containerName="main" containerID="cri-o://a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7" gracePeriod=45
Sep 10 03:05:58.675660 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.675600    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:05:58.803718 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.803683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tls-certs\") pod \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") "
Sep 10 03:05:58.803911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.803741    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tmp-dir\") pod \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") "
Sep 10 03:05:58.803911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.803760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-model-cache\") pod \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") "
Sep 10 03:05:58.803911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.803820    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5dqk\" (UniqueName: \"kubernetes.io/projected/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-kube-api-access-r5dqk\") pod \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") "
Sep 10 03:05:58.803911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.803852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-dshm\") pod \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") "
Sep 10 03:05:58.803911 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.803898    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-home\") pod \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\" (UID: \"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c\") "
Sep 10 03:05:58.804209 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.804052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" (UID: "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:05:58.804209 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.804070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" (UID: "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:05:58.804339 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.804239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-home" (OuterVolumeSpecName: "home") pod "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" (UID: "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:05:58.804380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.804365    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:58.804416 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.804384    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:58.804416 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.804397    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:58.805953 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.805932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" (UID: "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:05:58.806043 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.806002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-kube-api-access-r5dqk" (OuterVolumeSpecName: "kube-api-access-r5dqk") pod "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" (UID: "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c"). InnerVolumeSpecName "kube-api-access-r5dqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:05:58.806085 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.806037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-dshm" (OuterVolumeSpecName: "dshm") pod "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" (UID: "95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:05:58.904988 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.904956    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:58.904988 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.904982    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:58.904988 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:58.904994    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dqk\" (UniqueName: \"kubernetes.io/projected/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c-kube-api-access-r5dqk\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:05:59.427007 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.426967    2580 generic.go:356] "Generic (PLEG): container finished" podID="95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" containerID="a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7" exitCode=0
Sep 10 03:05:59.427445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.427013    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"
Sep 10 03:05:59.427445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.427049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm" event={"ID":"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c","Type":"ContainerDied","Data":"a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7"}
Sep 10 03:05:59.427445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.427097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm" event={"ID":"95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c","Type":"ContainerDied","Data":"5532544d4b677d17a7a2735065d1690cd7d71e3d85099495f4d4d04e55559cec"}
Sep 10 03:05:59.427445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.427117    2580 scope.go:117] "RemoveContainer" containerID="a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7"
Sep 10 03:05:59.436331 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.436315    2580 scope.go:117] "RemoveContainer" containerID="a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7"
Sep 10 03:05:59.436628 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:05:59.436610    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7\": container with ID starting with a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7 not found: ID does not exist" containerID="a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7"
Sep 10 03:05:59.436673 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.436637    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7"} err="failed to get container status \"a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7\": rpc error: code = NotFound desc = could not find container \"a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7\": container with ID starting with a57e8adc45a619a84c73cea116bd8aec69515fc012157a543b56ae734e41f4c7 not found: ID does not exist"
Sep 10 03:05:59.444356 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.444332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"]
Sep 10 03:05:59.448011 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:05:59.447990    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b0f4da/test-model-b0f4da-kserve-778c74d86c-9g8pm"]
Sep 10 03:06:01.325246 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:01.325212    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" path="/var/lib/kubelet/pods/95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c/volumes"
Sep 10 03:06:02.481401 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:02.481366    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"]
Sep 10 03:06:02.484361 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:02.484339    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0f4da-29816820-2vv7p"]
Sep 10 03:06:03.326408 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:03.326376    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf32dce-3382-48c2-baec-fca97d1a866e" path="/var/lib/kubelet/pods/adf32dce-3382-48c2-baec-fca97d1a866e/volumes"
Sep 10 03:06:18.116317 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:18.116237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:06:19.813536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:19.813500    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"]
Sep 10 03:06:19.814032 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:19.813739    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb" podUID="49c23839-197c-4004-a0ed-2541c16f2a5c" containerName="payload-processing" containerID="cri-o://faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5" gracePeriod=30
Sep 10 03:06:19.820049 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:19.819954    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"]
Sep 10 03:06:19.820287 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:19.820175    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh" podUID="ec566af6-700d-4672-a732-c7b0950db52d" containerName="payload-pre-processing" containerID="cri-o://b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426" gracePeriod=30
Sep 10 03:06:20.116523 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.116497    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 03:06:20.123853 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.123832    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 03:06:20.214196 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.214148    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvvj4\" (UniqueName: \"kubernetes.io/projected/ec566af6-700d-4672-a732-c7b0950db52d-kube-api-access-wvvj4\") pod \"ec566af6-700d-4672-a732-c7b0950db52d\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") "
Sep 10 03:06:20.214362 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.214214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7gjc\" (UniqueName: \"kubernetes.io/projected/49c23839-197c-4004-a0ed-2541c16f2a5c-kube-api-access-v7gjc\") pod \"49c23839-197c-4004-a0ed-2541c16f2a5c\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") "
Sep 10 03:06:20.214362 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.214277    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c23839-197c-4004-a0ed-2541c16f2a5c-plugins-config-volume\") pod \"49c23839-197c-4004-a0ed-2541c16f2a5c\" (UID: \"49c23839-197c-4004-a0ed-2541c16f2a5c\") "
Sep 10 03:06:20.214362 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.214321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec566af6-700d-4672-a732-c7b0950db52d-plugins-config-volume\") pod \"ec566af6-700d-4672-a732-c7b0950db52d\" (UID: \"ec566af6-700d-4672-a732-c7b0950db52d\") "
Sep 10 03:06:20.214709 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.214682    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c23839-197c-4004-a0ed-2541c16f2a5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49c23839-197c-4004-a0ed-2541c16f2a5c" (UID: "49c23839-197c-4004-a0ed-2541c16f2a5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:06:20.214798 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.214727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec566af6-700d-4672-a732-c7b0950db52d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec566af6-700d-4672-a732-c7b0950db52d" (UID: "ec566af6-700d-4672-a732-c7b0950db52d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:06:20.216231 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.216204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec566af6-700d-4672-a732-c7b0950db52d-kube-api-access-wvvj4" (OuterVolumeSpecName: "kube-api-access-wvvj4") pod "ec566af6-700d-4672-a732-c7b0950db52d" (UID: "ec566af6-700d-4672-a732-c7b0950db52d"). InnerVolumeSpecName "kube-api-access-wvvj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:06:20.216447 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.216431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c23839-197c-4004-a0ed-2541c16f2a5c-kube-api-access-v7gjc" (OuterVolumeSpecName: "kube-api-access-v7gjc") pod "49c23839-197c-4004-a0ed-2541c16f2a5c" (UID: "49c23839-197c-4004-a0ed-2541c16f2a5c"). InnerVolumeSpecName "kube-api-access-v7gjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:06:20.315593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.315564    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c23839-197c-4004-a0ed-2541c16f2a5c-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:06:20.315593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.315589    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec566af6-700d-4672-a732-c7b0950db52d-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:06:20.315593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.315600    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvvj4\" (UniqueName: \"kubernetes.io/projected/ec566af6-700d-4672-a732-c7b0950db52d-kube-api-access-wvvj4\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:06:20.315814 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.315609    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7gjc\" (UniqueName: \"kubernetes.io/projected/49c23839-197c-4004-a0ed-2541c16f2a5c-kube-api-access-v7gjc\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:06:20.552664 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.552626    2580 generic.go:356] "Generic (PLEG): container finished" podID="49c23839-197c-4004-a0ed-2541c16f2a5c" containerID="faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5" exitCode=0
Sep 10 03:06:20.552841 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.552675    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"
Sep 10 03:06:20.552841 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.552715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb" event={"ID":"49c23839-197c-4004-a0ed-2541c16f2a5c","Type":"ContainerDied","Data":"faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5"}
Sep 10 03:06:20.552841 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.552782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb" event={"ID":"49c23839-197c-4004-a0ed-2541c16f2a5c","Type":"ContainerDied","Data":"03c3ee572ec1fb8fe7c7caf21d82dc8d5c97db5093127437eba4112b18b30665"}
Sep 10 03:06:20.552841 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.552802    2580 scope.go:117] "RemoveContainer" containerID="faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5"
Sep 10 03:06:20.554060 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.554038    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec566af6-700d-4672-a732-c7b0950db52d" containerID="b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426" exitCode=0
Sep 10 03:06:20.554135 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.554075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh" event={"ID":"ec566af6-700d-4672-a732-c7b0950db52d","Type":"ContainerDied","Data":"b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426"}
Sep 10 03:06:20.554135 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.554100    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"
Sep 10 03:06:20.554135 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.554109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh" event={"ID":"ec566af6-700d-4672-a732-c7b0950db52d","Type":"ContainerDied","Data":"80758eb630b4c31e2ae5aa1209c672ccf4a829455767ab2a53c38d58c1d85599"}
Sep 10 03:06:20.561939 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.561920    2580 scope.go:117] "RemoveContainer" containerID="faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5"
Sep 10 03:06:20.562246 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:06:20.562227    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5\": container with ID starting with faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5 not found: ID does not exist" containerID="faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5"
Sep 10 03:06:20.562317 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.562253    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5"} err="failed to get container status \"faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5\": rpc error: code = NotFound desc = could not find container \"faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5\": container with ID starting with faeb3449a803e2dc12985d5123fe05b38b1908ff67c591b6a9fb328415eadcf5 not found: ID does not exist"
Sep 10 03:06:20.562317 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.562268    2580 scope.go:117] "RemoveContainer" containerID="b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426"
Sep 10 03:06:20.570600 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.570584    2580 scope.go:117] "RemoveContainer" containerID="b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426"
Sep 10 03:06:20.570836 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:06:20.570820    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426\": container with ID starting with b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426 not found: ID does not exist" containerID="b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426"
Sep 10 03:06:20.570900 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.570844    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426"} err="failed to get container status \"b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426\": rpc error: code = NotFound desc = could not find container \"b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426\": container with ID starting with b2cff161735769eefb2da098e3f329d842c1a6075d1694a32cd072465e7fb426 not found: ID does not exist"
Sep 10 03:06:20.579610 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.579588    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"]
Sep 10 03:06:20.583330 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.583310    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce5f9d-5974c8dc74-4jdjh"]
Sep 10 03:06:20.595895 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.595875    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"]
Sep 10 03:06:20.597934 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:20.597917    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce5f9d-6fd6ddbf5f-j9jvb"]
Sep 10 03:06:21.326482 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:21.326441    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c23839-197c-4004-a0ed-2541c16f2a5c" path="/var/lib/kubelet/pods/49c23839-197c-4004-a0ed-2541c16f2a5c/volumes"
Sep 10 03:06:21.326980 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:21.326942    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec566af6-700d-4672-a732-c7b0950db52d" path="/var/lib/kubelet/pods/ec566af6-700d-4672-a732-c7b0950db52d/volumes"
Sep 10 03:06:30.009384 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009350    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"]
Sep 10 03:06:30.009851 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009762    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec566af6-700d-4672-a732-c7b0950db52d" containerName="payload-pre-processing"
Sep 10 03:06:30.009851 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009773    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec566af6-700d-4672-a732-c7b0950db52d" containerName="payload-pre-processing"
Sep 10 03:06:30.009851 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009786    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c23839-197c-4004-a0ed-2541c16f2a5c" containerName="payload-processing"
Sep 10 03:06:30.009851 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009792    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c23839-197c-4004-a0ed-2541c16f2a5c" containerName="payload-processing"
Sep 10 03:06:30.009851 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009809    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" containerName="main"
Sep 10 03:06:30.009851 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009814    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" containerName="main"
Sep 10 03:06:30.010051 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009884    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95bd4ab9-5d5d-4c36-b14a-50d875c5cb1c" containerName="main"
Sep 10 03:06:30.010051 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009892    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c23839-197c-4004-a0ed-2541c16f2a5c" containerName="payload-processing"
Sep 10 03:06:30.010051 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.009900    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec566af6-700d-4672-a732-c7b0950db52d" containerName="payload-pre-processing"
Sep 10 03:06:30.014574 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.014556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.017514 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.017491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9b8682\"/\"openshift-service-ca.crt\""
Sep 10 03:06:30.017633 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.017546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9b8682\"/\"kube-root-ca.crt\""
Sep 10 03:06:30.018941 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.018920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9b8682\"/\"default-dockercfg-pt94g\""
Sep 10 03:06:30.019132 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.019093    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9b8682\"/\"noten-model-9b8682-kserve-self-signed-certs\""
Sep 10 03:06:30.022158 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.022133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"]
Sep 10 03:06:30.121912 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.121869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-model-cache\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.122233 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.122205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-home\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.122441 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.122425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-dshm\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.122600 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.122575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-tmp-dir\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.122748 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.122639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d837cb-c56a-4531-b326-3d5dc227f25c-tls-certs\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.122748 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.122700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxd6\" (UniqueName: \"kubernetes.io/projected/63d837cb-c56a-4531-b326-3d5dc227f25c-kube-api-access-kwxd6\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224127 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-home\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224346 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-dshm\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224346 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-tmp-dir\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224346 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d837cb-c56a-4531-b326-3d5dc227f25c-tls-certs\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224346 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxd6\" (UniqueName: \"kubernetes.io/projected/63d837cb-c56a-4531-b326-3d5dc227f25c-kube-api-access-kwxd6\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224575 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-model-cache\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224627 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-home\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224627 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-tmp-dir\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.224727 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.224711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-model-cache\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.226477 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.226460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-dshm\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.226791 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.226772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d837cb-c56a-4531-b326-3d5dc227f25c-tls-certs\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.232834 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.232817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxd6\" (UniqueName: \"kubernetes.io/projected/63d837cb-c56a-4531-b326-3d5dc227f25c-kube-api-access-kwxd6\") pod \"noten-model-9b8682-kserve-68557ffc5f-2wq9l\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") " pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.327602 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.327523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:30.606643 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:30.606618    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"]
Sep 10 03:06:30.608192 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:06:30.608152    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63d837cb_c56a_4531_b326_3d5dc227f25c.slice/crio-392e0ce2ed980de934269e3e5e1bcdb95c363a8293dd5644fe6ce86fdf49b799 WatchSource:0}: Error finding container 392e0ce2ed980de934269e3e5e1bcdb95c363a8293dd5644fe6ce86fdf49b799: Status 404 returned error can't find the container with id 392e0ce2ed980de934269e3e5e1bcdb95c363a8293dd5644fe6ce86fdf49b799
Sep 10 03:06:31.619592 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:31.619553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l" event={"ID":"63d837cb-c56a-4531-b326-3d5dc227f25c","Type":"ContainerStarted","Data":"a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74"}
Sep 10 03:06:31.619592 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:31.619595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l" event={"ID":"63d837cb-c56a-4531-b326-3d5dc227f25c","Type":"ContainerStarted","Data":"392e0ce2ed980de934269e3e5e1bcdb95c363a8293dd5644fe6ce86fdf49b799"}
Sep 10 03:06:31.639384 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:31.639334    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l" podStartSLOduration=2.63931905 podStartE2EDuration="2.63931905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:06:31.637327074 +0000 UTC m=+1188.787226771" watchObservedRunningTime="2026-09-10 03:06:31.63931905 +0000 UTC m=+1188.789218748"
Sep 10 03:06:40.328043 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:40.328007    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:40.328468 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:40.328057    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:40.340357 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:40.340335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:40.684337 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:40.684311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:06:43.507580 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:43.507543    2580 scope.go:117] "RemoveContainer" containerID="b972a8442d8acfacc215847683f8c3767cc96b5eef2fe2dc3671a9dff63cab5a"
Sep 10 03:06:43.516249 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:43.516217    2580 scope.go:117] "RemoveContainer" containerID="4df2488e76b471fd2c404138c2ab6f9cadd37754aae3c0d7c6cc2b4086428c38"
Sep 10 03:06:43.524412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:43.524386    2580 scope.go:117] "RemoveContainer" containerID="bd560ee19b30d1836cbc28261e47bcd46bdddfdcc718b43fcbceca906bf89d92"
Sep 10 03:06:43.810017 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:06:43.809916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:07:02.576012 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:02.575977    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"]
Sep 10 03:07:02.576591 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:02.576256    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy" containerID="cri-o://b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba" gracePeriod=30
Sep 10 03:07:04.459609 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:04.459576    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"]
Sep 10 03:07:04.460063 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:04.459828    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n" podUID="c98d2139-403e-4d58-89f9-f5e95640c379" containerName="istio-proxy" containerID="cri-o://dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56" gracePeriod=30
Sep 10 03:07:07.819628 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.819597    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 03:07:07.824540 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.824515    2580 generic.go:356] "Generic (PLEG): container finished" podID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerID="b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba" exitCode=0
Sep 10 03:07:07.824682 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.824560    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"
Sep 10 03:07:07.824682 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.824593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" event={"ID":"65b674bd-3112-4d6c-956b-91c5be7739f8","Type":"ContainerDied","Data":"b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba"}
Sep 10 03:07:07.824682 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.824648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj" event={"ID":"65b674bd-3112-4d6c-956b-91c5be7739f8","Type":"ContainerDied","Data":"4d7d0a69b2aeb8bbbba4bd32e62e5cd3ee492244d8ce8398d9237648a8f9f8fd"}
Sep 10 03:07:07.824682 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.824671    2580 scope.go:117] "RemoveContainer" containerID="b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba"
Sep 10 03:07:07.833124 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.833109    2580 scope.go:117] "RemoveContainer" containerID="93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82"
Sep 10 03:07:07.842101 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.842084    2580 scope.go:117] "RemoveContainer" containerID="b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba"
Sep 10 03:07:07.842459 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:07:07.842435    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba\": container with ID starting with b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba not found: ID does not exist" containerID="b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba"
Sep 10 03:07:07.842544 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.842471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba"} err="failed to get container status \"b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba\": rpc error: code = NotFound desc = could not find container \"b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba\": container with ID starting with b9bc1373ebf7d94bebba071142602c06efb7bcc06a17e51246556c2ab69f96ba not found: ID does not exist"
Sep 10 03:07:07.842544 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.842494    2580 scope.go:117] "RemoveContainer" containerID="93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82"
Sep 10 03:07:07.842774 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:07:07.842756    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82\": container with ID starting with 93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82 not found: ID does not exist" containerID="93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82"
Sep 10 03:07:07.842847 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.842780    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82"} err="failed to get container status \"93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82\": rpc error: code = NotFound desc = could not find container \"93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82\": container with ID starting with 93b6001ae8b9345335db86fc497fd07a3402a86eee725c43bb12add7e8b46c82 not found: ID does not exist"
Sep 10 03:07:07.921050 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-data\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-token\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-credential-socket\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-certs\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qgzj\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-kube-api-access-5qgzj\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-socket\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921232    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-podinfo\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921589 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-envoy\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921589 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:07.921589 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b674bd-3112-4d6c-956b-91c5be7739f8-istiod-ca-cert\") pod \"65b674bd-3112-4d6c-956b-91c5be7739f8\" (UID: \"65b674bd-3112-4d6c-956b-91c5be7739f8\") "
Sep 10 03:07:07.921589 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:07.921589 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:07.921833 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:07.921833 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921761    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:07.921833 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921778    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:07.921833 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921791    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:07.921833 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921803    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:07.922098 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.921862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b674bd-3112-4d6c-956b-91c5be7739f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:07:07.923492 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.923463    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:07.923601 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.923496    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:07:07.923601 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.923543    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-kube-api-access-5qgzj" (OuterVolumeSpecName: "kube-api-access-5qgzj") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "kube-api-access-5qgzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:07:07.923714 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:07.923633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65b674bd-3112-4d6c-956b-91c5be7739f8" (UID: "65b674bd-3112-4d6c-956b-91c5be7739f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:07:08.022826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.022793    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:08.022826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.022821    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qgzj\" (UniqueName: \"kubernetes.io/projected/65b674bd-3112-4d6c-956b-91c5be7739f8-kube-api-access-5qgzj\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:08.022826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.022830    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:08.023114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.022838    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b674bd-3112-4d6c-956b-91c5be7739f8-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:08.023114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.022847    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b674bd-3112-4d6c-956b-91c5be7739f8-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:08.150513 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.150480    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"]
Sep 10 03:07:08.153367 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.153342    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce5f9d-openshift-default-7557c9bb5b-7bcfj"]
Sep 10 03:07:08.790348 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.790314    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"]
Sep 10 03:07:08.790766 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.790754    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy"
Sep 10 03:07:08.790766 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.790768    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy"
Sep 10 03:07:08.790875 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.790844    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy"
Sep 10 03:07:08.795213 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.795197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:08.797959 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.797943    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-af7727\""
Sep 10 03:07:08.798037 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.798018    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-af7727-dockercfg-6n8ts\""
Sep 10 03:07:08.804892 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.804863    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"]
Sep 10 03:07:08.932190 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.932131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grv6\" (UniqueName: \"kubernetes.io/projected/7bc5140b-875d-42ba-a493-8524f424c2a5-kube-api-access-5grv6\") pod \"payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:08.932574 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:08.932310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc5140b-875d-42ba-a493-8524f424c2a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:09.033771 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.033732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc5140b-875d-42ba-a493-8524f424c2a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:09.033941 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.033817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grv6\" (UniqueName: \"kubernetes.io/projected/7bc5140b-875d-42ba-a493-8524f424c2a5-kube-api-access-5grv6\") pod \"payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:09.034441 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.034423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc5140b-875d-42ba-a493-8524f424c2a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:09.041927 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.041870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grv6\" (UniqueName: \"kubernetes.io/projected/7bc5140b-875d-42ba-a493-8524f424c2a5-kube-api-access-5grv6\") pod \"payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:09.106863 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.106830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:09.326075 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.325992    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" path="/var/lib/kubelet/pods/65b674bd-3112-4d6c-956b-91c5be7739f8/volumes"
Sep 10 03:07:09.377914 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.377885    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"]
Sep 10 03:07:09.379668 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:07:09.379645    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc5140b_875d_42ba_a493_8524f424c2a5.slice/crio-8631db82b4144e9904c8fe14d764c9ece50e59d26b271efa0d2c58a054c0f1bd WatchSource:0}: Error finding container 8631db82b4144e9904c8fe14d764c9ece50e59d26b271efa0d2c58a054c0f1bd: Status 404 returned error can't find the container with id 8631db82b4144e9904c8fe14d764c9ece50e59d26b271efa0d2c58a054c0f1bd
Sep 10 03:07:09.706977 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.706952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:07:09.818934 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.818903    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"]
Sep 10 03:07:09.839753 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.839640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr" event={"ID":"7bc5140b-875d-42ba-a493-8524f424c2a5","Type":"ContainerStarted","Data":"d6fe746bcf9ed4bef9f7c320aeeb1a33e4869161abc6c67cfca9906818613e91"}
Sep 10 03:07:09.839753 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.839695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr" event={"ID":"7bc5140b-875d-42ba-a493-8524f424c2a5","Type":"ContainerStarted","Data":"8631db82b4144e9904c8fe14d764c9ece50e59d26b271efa0d2c58a054c0f1bd"}
Sep 10 03:07:09.839978 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.839905    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:09.841255 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.841231    2580 generic.go:356] "Generic (PLEG): container finished" podID="c98d2139-403e-4d58-89f9-f5e95640c379" containerID="dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56" exitCode=0
Sep 10 03:07:09.841406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.841276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"
Sep 10 03:07:09.841406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.841292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n" event={"ID":"c98d2139-403e-4d58-89f9-f5e95640c379","Type":"ContainerDied","Data":"dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56"}
Sep 10 03:07:09.841406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.841334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n" event={"ID":"c98d2139-403e-4d58-89f9-f5e95640c379","Type":"ContainerDied","Data":"b021b465ea09caff71e62ae95c23494a7b90e9edbb5625db8cf14a77a3b2c0fd"}
Sep 10 03:07:09.841406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.841361    2580 scope.go:117] "RemoveContainer" containerID="dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56"
Sep 10 03:07:09.842448 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-credential-socket\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842526 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-istio-token\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842526 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842510    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-socket\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842623 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-certs\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842623 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842587    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-data\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842719 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-envoy\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842719 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9dll\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-kube-api-access-r9dll\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842719 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:09.842870 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98d2139-403e-4d58-89f9-f5e95640c379-istiod-ca-cert\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842870 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98d2139-403e-4d58-89f9-f5e95640c379-istio-podinfo\") pod \"c98d2139-403e-4d58-89f9-f5e95640c379\" (UID: \"c98d2139-403e-4d58-89f9-f5e95640c379\") "
Sep 10 03:07:09.842989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:09.842989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.842898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:09.843158 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.843140    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.843398 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.843329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-data" (OuterVolumeSpecName: "istio-data") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:09.843398 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.843345    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.843398 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.843364    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.843587 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.843495    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98d2139-403e-4d58-89f9-f5e95640c379-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:07:09.845447 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.845424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-istio-token" (OuterVolumeSpecName: "istio-token") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:07:09.845559 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.845538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:09.845611 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.845574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-kube-api-access-r9dll" (OuterVolumeSpecName: "kube-api-access-r9dll") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "kube-api-access-r9dll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:07:09.845611 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.845583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c98d2139-403e-4d58-89f9-f5e95640c379-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c98d2139-403e-4d58-89f9-f5e95640c379" (UID: "c98d2139-403e-4d58-89f9-f5e95640c379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:07:09.851226 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.851210    2580 scope.go:117] "RemoveContainer" containerID="dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56"
Sep 10 03:07:09.851507 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:07:09.851488    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56\": container with ID starting with dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56 not found: ID does not exist" containerID="dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56"
Sep 10 03:07:09.851581 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.851519    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56"} err="failed to get container status \"dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56\": rpc error: code = NotFound desc = could not find container \"dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56\": container with ID starting with dd4c80754e8437d2728a7aed94c758d80e7e6ff9aadf7e5089d3b955038c4a56 not found: ID does not exist"
Sep 10 03:07:09.859579 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.859533    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr" podStartSLOduration=1.859516128 podStartE2EDuration="1.859516128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:09.856383713 +0000 UTC m=+1227.006283436" watchObservedRunningTime="2026-09-10 03:07:09.859516128 +0000 UTC m=+1227.009415830"
Sep 10 03:07:09.944714 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.944679    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.944714 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.944711    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9dll\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-kube-api-access-r9dll\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.945078 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.944727    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98d2139-403e-4d58-89f9-f5e95640c379-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.945078 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.944742    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98d2139-403e-4d58-89f9-f5e95640c379-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.945078 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.944755    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98d2139-403e-4d58-89f9-f5e95640c379-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:09.945078 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:09.944767    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98d2139-403e-4d58-89f9-f5e95640c379-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:10.165088 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:10.165061    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"]
Sep 10 03:07:10.168311 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:10.168286    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-9b8682-openshift-default-568bf6c65d-2dj5n"]
Sep 10 03:07:11.325191 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:11.325131    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c98d2139-403e-4d58-89f9-f5e95640c379" path="/var/lib/kubelet/pods/c98d2139-403e-4d58-89f9-f5e95640c379/volumes"
Sep 10 03:07:14.122735 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:14.122703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:07:20.850713 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:20.850685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:07:24.839828 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:24.839784    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l" podUID="63d837cb-c56a-4531-b326-3d5dc227f25c" containerName="main" containerID="cri-o://a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74" gracePeriod=45
Sep 10 03:07:25.082685 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.082660    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:07:25.192837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.192794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-model-cache\") pod \"63d837cb-c56a-4531-b326-3d5dc227f25c\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") "
Sep 10 03:07:25.193024 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.192857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d837cb-c56a-4531-b326-3d5dc227f25c-tls-certs\") pod \"63d837cb-c56a-4531-b326-3d5dc227f25c\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") "
Sep 10 03:07:25.193024 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.192899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-tmp-dir\") pod \"63d837cb-c56a-4531-b326-3d5dc227f25c\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") "
Sep 10 03:07:25.193142 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwxd6\" (UniqueName: \"kubernetes.io/projected/63d837cb-c56a-4531-b326-3d5dc227f25c-kube-api-access-kwxd6\") pod \"63d837cb-c56a-4531-b326-3d5dc227f25c\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") "
Sep 10 03:07:25.193142 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-model-cache" (OuterVolumeSpecName: "model-cache") pod "63d837cb-c56a-4531-b326-3d5dc227f25c" (UID: "63d837cb-c56a-4531-b326-3d5dc227f25c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:25.193142 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63d837cb-c56a-4531-b326-3d5dc227f25c" (UID: "63d837cb-c56a-4531-b326-3d5dc227f25c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:25.193142 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-home\") pod \"63d837cb-c56a-4531-b326-3d5dc227f25c\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") "
Sep 10 03:07:25.193370 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193159    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-dshm\") pod \"63d837cb-c56a-4531-b326-3d5dc227f25c\" (UID: \"63d837cb-c56a-4531-b326-3d5dc227f25c\") "
Sep 10 03:07:25.193370 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-home" (OuterVolumeSpecName: "home") pod "63d837cb-c56a-4531-b326-3d5dc227f25c" (UID: "63d837cb-c56a-4531-b326-3d5dc227f25c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:25.193560 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193539    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:25.193560 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193559    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:25.193755 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.193596    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:25.195095 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.195069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d837cb-c56a-4531-b326-3d5dc227f25c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63d837cb-c56a-4531-b326-3d5dc227f25c" (UID: "63d837cb-c56a-4531-b326-3d5dc227f25c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:07:25.195095 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.195082    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d837cb-c56a-4531-b326-3d5dc227f25c-kube-api-access-kwxd6" (OuterVolumeSpecName: "kube-api-access-kwxd6") pod "63d837cb-c56a-4531-b326-3d5dc227f25c" (UID: "63d837cb-c56a-4531-b326-3d5dc227f25c"). InnerVolumeSpecName "kube-api-access-kwxd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:07:25.195379 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.195069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-dshm" (OuterVolumeSpecName: "dshm") pod "63d837cb-c56a-4531-b326-3d5dc227f25c" (UID: "63d837cb-c56a-4531-b326-3d5dc227f25c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:07:25.294329 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.294292    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxd6\" (UniqueName: \"kubernetes.io/projected/63d837cb-c56a-4531-b326-3d5dc227f25c-kube-api-access-kwxd6\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:25.294329 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.294324    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d837cb-c56a-4531-b326-3d5dc227f25c-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:25.294329 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.294333    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d837cb-c56a-4531-b326-3d5dc227f25c-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:07:25.934023 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.933978    2580 generic.go:356] "Generic (PLEG): container finished" podID="63d837cb-c56a-4531-b326-3d5dc227f25c" containerID="a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74" exitCode=0
Sep 10 03:07:25.934023 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.934023    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"
Sep 10 03:07:25.934515 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.934057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l" event={"ID":"63d837cb-c56a-4531-b326-3d5dc227f25c","Type":"ContainerDied","Data":"a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74"}
Sep 10 03:07:25.934515 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.934102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l" event={"ID":"63d837cb-c56a-4531-b326-3d5dc227f25c","Type":"ContainerDied","Data":"392e0ce2ed980de934269e3e5e1bcdb95c363a8293dd5644fe6ce86fdf49b799"}
Sep 10 03:07:25.934515 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.934122    2580 scope.go:117] "RemoveContainer" containerID="a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74"
Sep 10 03:07:25.942996 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.942979    2580 scope.go:117] "RemoveContainer" containerID="a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74"
Sep 10 03:07:25.943268 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:07:25.943250    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74\": container with ID starting with a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74 not found: ID does not exist" containerID="a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74"
Sep 10 03:07:25.943336 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.943282    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74"} err="failed to get container status \"a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74\": rpc error: code = NotFound desc = could not find container \"a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74\": container with ID starting with a3ac7359b1dbbf40d25abd5510badcb41ba74d2581da7e0e0429924f45f55c74 not found: ID does not exist"
Sep 10 03:07:25.951541 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.951517    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"]
Sep 10 03:07:25.956529 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:25.956506    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-9b8682/noten-model-9b8682-kserve-68557ffc5f-2wq9l"]
Sep 10 03:07:27.325257 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:27.325226    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d837cb-c56a-4531-b326-3d5dc227f25c" path="/var/lib/kubelet/pods/63d837cb-c56a-4531-b326-3d5dc227f25c/volumes"
Sep 10 03:07:32.516531 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.516496    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"]
Sep 10 03:07:32.517122 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517106    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy"
Sep 10 03:07:32.517199 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517125    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy"
Sep 10 03:07:32.517199 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517148    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c98d2139-403e-4d58-89f9-f5e95640c379" containerName="istio-proxy"
Sep 10 03:07:32.517199 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517157    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c98d2139-403e-4d58-89f9-f5e95640c379" containerName="istio-proxy"
Sep 10 03:07:32.517310 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517211    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d837cb-c56a-4531-b326-3d5dc227f25c" containerName="main"
Sep 10 03:07:32.517310 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517220    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d837cb-c56a-4531-b326-3d5dc227f25c" containerName="main"
Sep 10 03:07:32.517310 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517308    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d837cb-c56a-4531-b326-3d5dc227f25c" containerName="main"
Sep 10 03:07:32.517406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517318    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c98d2139-403e-4d58-89f9-f5e95640c379" containerName="istio-proxy"
Sep 10 03:07:32.517406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.517329    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b674bd-3112-4d6c-956b-91c5be7739f8" containerName="istio-proxy"
Sep 10 03:07:32.522077 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.522058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.524985 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.524964    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7daa14-openshift-default-dockercfg-99gkd\""
Sep 10 03:07:32.531923 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.531900    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"]
Sep 10 03:07:32.665064 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-socket\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665289 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-credential-socket\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665289 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-token\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665289 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-data\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665289 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-certs\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jj7\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-kube-api-access-w7jj7\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-podinfo\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-envoy\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.665442 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.665413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istiod-ca-cert\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766389 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-envoy\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766389 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istiod-ca-cert\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766680 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-socket\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766680 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-credential-socket\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766680 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-token\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766680 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-data\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766897 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-certs\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766897 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jj7\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-kube-api-access-w7jj7\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766897 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-podinfo\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.766897 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-socket\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.767095 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-credential-socket\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.767095 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.766948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-data\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.767239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.767115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-certs\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.767239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.767141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istiod-ca-cert\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.768627 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.768605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-envoy\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.769020 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.769003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-podinfo\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.774311 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.774289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-token\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.774412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.774364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jj7\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-kube-api-access-w7jj7\") pod \"e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") " pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:32.834005 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:32.833979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:33.108803 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:33.108773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"]
Sep 10 03:07:33.110420 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:07:33.110393    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3b764b_ceea_4f90_a810_d2dc9bfb9449.slice/crio-c5c57e1b3d0cf4ec7a0631312be3ecc79f45be62e548ceccf974a5c624177d51 WatchSource:0}: Error finding container c5c57e1b3d0cf4ec7a0631312be3ecc79f45be62e548ceccf974a5c624177d51: Status 404 returned error can't find the container with id c5c57e1b3d0cf4ec7a0631312be3ecc79f45be62e548ceccf974a5c624177d51
Sep 10 03:07:33.112624 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:33.112595    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:07:33.112704 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:33.112655    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:07:33.112704 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:33.112683    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:07:33.977614 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:33.977575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" event={"ID":"6b3b764b-ceea-4f90-a810-d2dc9bfb9449","Type":"ContainerStarted","Data":"be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a"}
Sep 10 03:07:33.978081 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:33.977633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" event={"ID":"6b3b764b-ceea-4f90-a810-d2dc9bfb9449","Type":"ContainerStarted","Data":"c5c57e1b3d0cf4ec7a0631312be3ecc79f45be62e548ceccf974a5c624177d51"}
Sep 10 03:07:34.002846 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:34.002800    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" podStartSLOduration=2.002784815 podStartE2EDuration="2.002784815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:33.999944118 +0000 UTC m=+1251.149843828" watchObservedRunningTime="2026-09-10 03:07:34.002784815 +0000 UTC m=+1251.152684511"
Sep 10 03:07:34.834336 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:34.834296    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:34.839027 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:34.839000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:34.982836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:34.982801    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:34.983635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:34.983617    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:37.854836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.854803    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"]
Sep 10 03:07:37.858593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.858573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:37.861369 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.861346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7daa14-dockercfg-k8qm8\""
Sep 10 03:07:37.861369 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.861366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7daa14\""
Sep 10 03:07:37.870231 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.870203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"]
Sep 10 03:07:37.879472 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.879453    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"]
Sep 10 03:07:37.883299 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.883278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:37.893143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:37.893124    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"]
Sep 10 03:07:38.020551 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.020516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b470f1-aa77-40ad-a508-fb9d048db979-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:38.020720 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.020619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkrw\" (UniqueName: \"kubernetes.io/projected/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-kube-api-access-2tkrw\") pod \"payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:38.020768 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.020718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qg2\" (UniqueName: \"kubernetes.io/projected/67b470f1-aa77-40ad-a508-fb9d048db979-kube-api-access-q5qg2\") pod \"payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:38.020768 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.020763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:38.122421 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.122336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qg2\" (UniqueName: \"kubernetes.io/projected/67b470f1-aa77-40ad-a508-fb9d048db979-kube-api-access-q5qg2\") pod \"payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:38.122421 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.122381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:38.122658 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.122441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b470f1-aa77-40ad-a508-fb9d048db979-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:38.122658 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.122504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkrw\" (UniqueName: \"kubernetes.io/projected/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-kube-api-access-2tkrw\") pod \"payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:38.123154 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.123131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:38.123247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.123140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b470f1-aa77-40ad-a508-fb9d048db979-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:38.130723 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.130697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkrw\" (UniqueName: \"kubernetes.io/projected/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-kube-api-access-2tkrw\") pod \"payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:38.130821 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.130702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qg2\" (UniqueName: \"kubernetes.io/projected/67b470f1-aa77-40ad-a508-fb9d048db979-kube-api-access-q5qg2\") pod \"payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:38.169581 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.169558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:38.195025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.195003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:38.440530 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.440506    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"]
Sep 10 03:07:38.442200 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:07:38.442154    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6a7f0f_b2d3_45b1_a215_6001a2072ea6.slice/crio-3772fb2d5d91c27b8151f3d58169f5159cd701ff6ce6dcbb025c17f89fec7b64 WatchSource:0}: Error finding container 3772fb2d5d91c27b8151f3d58169f5159cd701ff6ce6dcbb025c17f89fec7b64: Status 404 returned error can't find the container with id 3772fb2d5d91c27b8151f3d58169f5159cd701ff6ce6dcbb025c17f89fec7b64
Sep 10 03:07:38.473672 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:38.473648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"]
Sep 10 03:07:38.485239 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:07:38.485211    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b470f1_aa77_40ad_a508_fb9d048db979.slice/crio-5eaf56e2d73537b8aa68f7f5dbb173271ac7bb8b7647f661056e74496db68ae6 WatchSource:0}: Error finding container 5eaf56e2d73537b8aa68f7f5dbb173271ac7bb8b7647f661056e74496db68ae6: Status 404 returned error can't find the container with id 5eaf56e2d73537b8aa68f7f5dbb173271ac7bb8b7647f661056e74496db68ae6
Sep 10 03:07:39.008854 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.008817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv" event={"ID":"67b470f1-aa77-40ad-a508-fb9d048db979","Type":"ContainerStarted","Data":"127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52"}
Sep 10 03:07:39.009312 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.008880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv" event={"ID":"67b470f1-aa77-40ad-a508-fb9d048db979","Type":"ContainerStarted","Data":"5eaf56e2d73537b8aa68f7f5dbb173271ac7bb8b7647f661056e74496db68ae6"}
Sep 10 03:07:39.009312 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.008935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:39.010569 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.010543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr" event={"ID":"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6","Type":"ContainerStarted","Data":"fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6"}
Sep 10 03:07:39.010687 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.010586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:39.010687 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.010602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr" event={"ID":"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6","Type":"ContainerStarted","Data":"3772fb2d5d91c27b8151f3d58169f5159cd701ff6ce6dcbb025c17f89fec7b64"}
Sep 10 03:07:39.025745 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.025698    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv" podStartSLOduration=2.025685974 podStartE2EDuration="2.025685974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:39.02382421 +0000 UTC m=+1256.173723907" watchObservedRunningTime="2026-09-10 03:07:39.025685974 +0000 UTC m=+1256.175585724"
Sep 10 03:07:39.038727 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:39.038689    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr" podStartSLOduration=2.038676247 podStartE2EDuration="2.038676247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:39.037856766 +0000 UTC m=+1256.187756467" watchObservedRunningTime="2026-09-10 03:07:39.038676247 +0000 UTC m=+1256.188575942"
Sep 10 03:07:43.564091 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:43.564062    2580 scope.go:117] "RemoveContainer" containerID="ba16bbe54f678882660a0e9f443cf9a0fdbd4b089840ae5d720264604cc63055"
Sep 10 03:07:43.572154 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:43.572137    2580 scope.go:117] "RemoveContainer" containerID="c007a35043ff04945663c0a4e9324912f4299ce188912c56d97a49faa1ac8235"
Sep 10 03:07:46.619385 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:46.619349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:07:50.021681 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:50.021647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:07:50.023178 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:50.022577    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:07:57.110573 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:57.110508    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerID="be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a" exitCode=0
Sep 10 03:07:57.110913 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:57.110583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" event={"ID":"6b3b764b-ceea-4f90-a810-d2dc9bfb9449","Type":"ContainerDied","Data":"be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a"}
Sep 10 03:07:57.111053 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:57.111037    2580 scope.go:117] "RemoveContainer" containerID="be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a"
Sep 10 03:07:57.112327 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:57.112304    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:07:57.112401 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:57.112365    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:07:57.112401 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:57.112394    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:07:58.122337 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:58.122288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" event={"ID":"6b3b764b-ceea-4f90-a810-d2dc9bfb9449","Type":"ContainerStarted","Data":"1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93"}
Sep 10 03:07:58.834923 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:58.834884    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:07:58.835150 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:58.835125    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 03:07:58.835227 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:58.835207    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 03:07:59.834660 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:59.834619    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 03:07:59.835037 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:07:59.834692    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 03:08:00.834423 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:00.834379    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 03:08:00.834638 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:00.834450    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 03:08:01.834777 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:01.834739    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 03:08:01.835190 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:01.834807    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 03:08:02.834514 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:02.834484    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 03:08:02.834719 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:02.834551    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 03:08:03.838457 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:03.838426    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:08:03.838816 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:03.838697    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:08:03.839386 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:03.839368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:08:17.327102 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:17.327072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:08:38.401068 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.401034    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"]
Sep 10 03:08:38.403909 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.403892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.406671 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.406651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"kube-root-ca.crt\""
Sep 10 03:08:38.408049 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.408031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"auth-test-model-af7727-kserve-self-signed-certs\""
Sep 10 03:08:38.408049 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.408042    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"default-dockercfg-l2qbq\""
Sep 10 03:08:38.408189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.408133    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"openshift-service-ca.crt\""
Sep 10 03:08:38.415385 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.415365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"]
Sep 10 03:08:38.536475 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.536448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-home\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.536629 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.536511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-dshm\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.536629 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.536539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55h8\" (UniqueName: \"kubernetes.io/projected/4f9fedd4-b928-40a6-853f-858b180cc714-kube-api-access-z55h8\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.536629 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.536560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-tmp-dir\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.536629 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.536615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-model-cache\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.536769 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.536645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9fedd4-b928-40a6-853f-858b180cc714-tls-certs\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638394 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-model-cache\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9fedd4-b928-40a6-853f-858b180cc714-tls-certs\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-home\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-dshm\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55h8\" (UniqueName: \"kubernetes.io/projected/4f9fedd4-b928-40a6-853f-858b180cc714-kube-api-access-z55h8\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638788 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-tmp-dir\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638852 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-model-cache\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.638910 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.638851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-home\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.639028 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.639011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-tmp-dir\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.640637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.640615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-dshm\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.640985 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.640967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9fedd4-b928-40a6-853f-858b180cc714-tls-certs\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.647187 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.647148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55h8\" (UniqueName: \"kubernetes.io/projected/4f9fedd4-b928-40a6-853f-858b180cc714-kube-api-access-z55h8\") pod \"auth-test-model-af7727-kserve-58b759fbc5-rxf5k\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") " pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.714218 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.714122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:38.987478 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:38.987451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"]
Sep 10 03:08:38.988503 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:08:38.988477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f9fedd4_b928_40a6_853f_858b180cc714.slice/crio-c2b0e8e92a4bfdb3cf21b690e35ac0d660d25c196c4b9d9b365e7e0b748165ee WatchSource:0}: Error finding container c2b0e8e92a4bfdb3cf21b690e35ac0d660d25c196c4b9d9b365e7e0b748165ee: Status 404 returned error can't find the container with id c2b0e8e92a4bfdb3cf21b690e35ac0d660d25c196c4b9d9b365e7e0b748165ee
Sep 10 03:08:39.348954 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:39.348918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k" event={"ID":"4f9fedd4-b928-40a6-853f-858b180cc714","Type":"ContainerStarted","Data":"09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909"}
Sep 10 03:08:39.349119 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:39.348970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k" event={"ID":"4f9fedd4-b928-40a6-853f-858b180cc714","Type":"ContainerStarted","Data":"c2b0e8e92a4bfdb3cf21b690e35ac0d660d25c196c4b9d9b365e7e0b748165ee"}
Sep 10 03:08:39.367363 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:39.367316    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k" podStartSLOduration=1.36730143 podStartE2EDuration="1.36730143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:08:39.365575358 +0000 UTC m=+1316.515475058" watchObservedRunningTime="2026-09-10 03:08:39.36730143 +0000 UTC m=+1316.517201162"
Sep 10 03:08:42.925515 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:42.925477    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:08:48.714663 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:48.714625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:48.715134 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:48.714686    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:48.728058 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:48.728034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:49.413221 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:49.413178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:08:54.607420 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.607388    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"]
Sep 10 03:08:54.610236 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.610216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.613700 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.613684    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"unconfigured-w4-e6e6b8-kserve-self-signed-certs\""
Sep 10 03:08:54.621662 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.621632    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"]
Sep 10 03:08:54.703181 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.703132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqz5x\" (UniqueName: \"kubernetes.io/projected/40ef286d-ed76-4d2f-9457-6257d364a2ed-kube-api-access-wqz5x\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.703380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.703209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef286d-ed76-4d2f-9457-6257d364a2ed-tls-certs\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.703380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.703237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-tmp-dir\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.703380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.703262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-model-cache\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.703380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.703356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-dshm\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.703578 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.703416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-home\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.806815 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.806775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqz5x\" (UniqueName: \"kubernetes.io/projected/40ef286d-ed76-4d2f-9457-6257d364a2ed-kube-api-access-wqz5x\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.807100 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.807078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef286d-ed76-4d2f-9457-6257d364a2ed-tls-certs\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.807262 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.807245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-tmp-dir\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.807389 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.807375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-model-cache\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.807599 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.807582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-dshm\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.807784 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.807769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-home\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.807999 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.807971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-tmp-dir\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.811254 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.810914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-dshm\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.811254 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.811214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-home\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.812192 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.811704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-model-cache\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.817566 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.817507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqz5x\" (UniqueName: \"kubernetes.io/projected/40ef286d-ed76-4d2f-9457-6257d364a2ed-kube-api-access-wqz5x\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.824615 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.824557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef286d-ed76-4d2f-9457-6257d364a2ed-tls-certs\") pod \"unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") " pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:54.919157 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:54.919053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:08:55.188480 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:55.188451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"]
Sep 10 03:08:55.191107 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:08:55.191079    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40ef286d_ed76_4d2f_9457_6257d364a2ed.slice/crio-29f9e57ee5c35ce09387117391cb05d241dcc746c58cd01a1bc8c905fff922f5 WatchSource:0}: Error finding container 29f9e57ee5c35ce09387117391cb05d241dcc746c58cd01a1bc8c905fff922f5: Status 404 returned error can't find the container with id 29f9e57ee5c35ce09387117391cb05d241dcc746c58cd01a1bc8c905fff922f5
Sep 10 03:08:55.436637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:55.436600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc" event={"ID":"40ef286d-ed76-4d2f-9457-6257d364a2ed","Type":"ContainerStarted","Data":"f89905ac625505062f130f75beb7544072bf2de34577f163af0edb169c141d72"}
Sep 10 03:08:55.436637 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:55.436642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc" event={"ID":"40ef286d-ed76-4d2f-9457-6257d364a2ed","Type":"ContainerStarted","Data":"29f9e57ee5c35ce09387117391cb05d241dcc746c58cd01a1bc8c905fff922f5"}
Sep 10 03:08:55.458691 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:08:55.458586    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc" podStartSLOduration=1.4585671740000001 podStartE2EDuration="1.458567174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:08:55.456260106 +0000 UTC m=+1332.606159802" watchObservedRunningTime="2026-09-10 03:08:55.458567174 +0000 UTC m=+1332.608466909"
Sep 10 03:09:04.919772 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:09:04.919737    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:09:04.920197 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:09:04.919790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:09:04.932418 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:09:04.932395    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:09:05.501803 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:09:05.501777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:09:12.523372 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:09:12.523289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:09:42.719002 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:09:42.718967    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:10:21.621584 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:10:21.621548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:10:53.620068 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:10:53.620029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:11:21.716518 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:21.716479    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"]
Sep 10 03:11:21.723338 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:21.723303    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"]
Sep 10 03:11:21.755567 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:21.755529    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"]
Sep 10 03:11:33.265972 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:33.265924    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"]
Sep 10 03:11:33.269196 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:33.269159    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-097e16-29816820-65nz8"]
Sep 10 03:11:33.328860 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:33.328822    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="468ec338-2565-4c37-8616-8a75622b08d7" path="/var/lib/kubelet/pods/468ec338-2565-4c37-8616-8a75622b08d7/volumes"
Sep 10 03:11:36.738597 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:36.738545    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6" podUID="55b16e80-04b5-4743-91e2-5c34e810133a" containerName="main" containerID="cri-o://78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c" gracePeriod=45
Sep 10 03:11:36.744435 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:36.744376    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw" podUID="5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" containerName="main" containerID="cri-o://fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598" gracePeriod=45
Sep 10 03:11:36.776745 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:36.776703    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w" podUID="cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" containerName="main" containerID="cri-o://688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838" gracePeriod=45
Sep 10 03:11:37.061523 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.061491    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:11:37.079387 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.079354    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 03:11:37.089416 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.089384    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:11:37.154920 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.154884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f7qq\" (UniqueName: \"kubernetes.io/projected/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-kube-api-access-2f7qq\") pod \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") "
Sep 10 03:11:37.155116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.154948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b16e80-04b5-4743-91e2-5c34e810133a-tls-certs\") pod \"55b16e80-04b5-4743-91e2-5c34e810133a\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") "
Sep 10 03:11:37.155116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.154973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjrng\" (UniqueName: \"kubernetes.io/projected/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-kube-api-access-pjrng\") pod \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") "
Sep 10 03:11:37.155116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.154990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4k9h\" (UniqueName: \"kubernetes.io/projected/55b16e80-04b5-4743-91e2-5c34e810133a-kube-api-access-n4k9h\") pod \"55b16e80-04b5-4743-91e2-5c34e810133a\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") "
Sep 10 03:11:37.155116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-dshm\") pod \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") "
Sep 10 03:11:37.155116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-model-cache\") pod \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") "
Sep 10 03:11:37.155116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155084    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-home\") pod \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") "
Sep 10 03:11:37.155116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-model-cache\") pod \"55b16e80-04b5-4743-91e2-5c34e810133a\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-home\") pod \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tmp-dir\") pod \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-tmp-dir\") pod \"55b16e80-04b5-4743-91e2-5c34e810133a\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155260    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tmp-dir\") pod \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-model-cache\") pod \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-dshm\") pod \"55b16e80-04b5-4743-91e2-5c34e810133a\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155352    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tls-certs\") pod \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\" (UID: \"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-dshm\") pod \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-home\") pod \"55b16e80-04b5-4743-91e2-5c34e810133a\" (UID: \"55b16e80-04b5-4743-91e2-5c34e810133a\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tls-certs\") pod \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\" (UID: \"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab\") "
Sep 10 03:11:37.155546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155501    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" (UID: "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.156075 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.155972    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.157155 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.157023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" (UID: "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.157314 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.157192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" (UID: "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.157669 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.157425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-home" (OuterVolumeSpecName: "home") pod "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" (UID: "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.157669 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.157651    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-model-cache" (OuterVolumeSpecName: "model-cache") pod "55b16e80-04b5-4743-91e2-5c34e810133a" (UID: "55b16e80-04b5-4743-91e2-5c34e810133a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.157937 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.157877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-home" (OuterVolumeSpecName: "home") pod "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" (UID: "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.158413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.158187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" (UID: "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.158413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.158200    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55b16e80-04b5-4743-91e2-5c34e810133a" (UID: "55b16e80-04b5-4743-91e2-5c34e810133a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.158413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.158255    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-kube-api-access-2f7qq" (OuterVolumeSpecName: "kube-api-access-2f7qq") pod "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" (UID: "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab"). InnerVolumeSpecName "kube-api-access-2f7qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:37.158413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.158298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b16e80-04b5-4743-91e2-5c34e810133a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55b16e80-04b5-4743-91e2-5c34e810133a" (UID: "55b16e80-04b5-4743-91e2-5c34e810133a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:11:37.158711 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.158445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-home" (OuterVolumeSpecName: "home") pod "55b16e80-04b5-4743-91e2-5c34e810133a" (UID: "55b16e80-04b5-4743-91e2-5c34e810133a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.160246 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.160216    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-kube-api-access-pjrng" (OuterVolumeSpecName: "kube-api-access-pjrng") pod "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" (UID: "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37"). InnerVolumeSpecName "kube-api-access-pjrng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:37.160872 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.160827    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-dshm" (OuterVolumeSpecName: "dshm") pod "55b16e80-04b5-4743-91e2-5c34e810133a" (UID: "55b16e80-04b5-4743-91e2-5c34e810133a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.160975 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.160897    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" (UID: "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:11:37.161043 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.160991    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" (UID: "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:11:37.161101 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.161052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-dshm" (OuterVolumeSpecName: "dshm") pod "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" (UID: "cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.161215 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.161194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b16e80-04b5-4743-91e2-5c34e810133a-kube-api-access-n4k9h" (OuterVolumeSpecName: "kube-api-access-n4k9h") pod "55b16e80-04b5-4743-91e2-5c34e810133a" (UID: "55b16e80-04b5-4743-91e2-5c34e810133a"). InnerVolumeSpecName "kube-api-access-n4k9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:37.161813 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.161795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-dshm" (OuterVolumeSpecName: "dshm") pod "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" (UID: "5bea1e32-b5c9-4f76-8bc2-894f24cf3a37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.256949    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.256979    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.256989    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.256997    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257004    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257012    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257020    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f7qq\" (UniqueName: \"kubernetes.io/projected/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-kube-api-access-2f7qq\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257029    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b16e80-04b5-4743-91e2-5c34e810133a-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257037    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjrng\" (UniqueName: \"kubernetes.io/projected/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-kube-api-access-pjrng\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257047    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k9h\" (UniqueName: \"kubernetes.io/projected/55b16e80-04b5-4743-91e2-5c34e810133a-kube-api-access-n4k9h\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257055    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257064    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257071    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257079    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257085    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257092    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b16e80-04b5-4743-91e2-5c34e810133a-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.257464 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.257099    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.343277 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.343248    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" containerID="688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838" exitCode=0
Sep 10 03:11:37.343372 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.343296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"
Sep 10 03:11:37.343372 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.343326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w" event={"ID":"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab","Type":"ContainerDied","Data":"688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838"}
Sep 10 03:11:37.343480 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.343372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w" event={"ID":"cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab","Type":"ContainerDied","Data":"294c5791c6e2619fc0558d07a8e4d1ff843be46f076eadb2aaace8a0c6c4fd0a"}
Sep 10 03:11:37.343480 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.343397    2580 scope.go:117] "RemoveContainer" containerID="688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838"
Sep 10 03:11:37.344813 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.344788    2580 generic.go:356] "Generic (PLEG): container finished" podID="5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" containerID="fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598" exitCode=0
Sep 10 03:11:37.344924 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.344833    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"
Sep 10 03:11:37.344924 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.344865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw" event={"ID":"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37","Type":"ContainerDied","Data":"fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598"}
Sep 10 03:11:37.344924 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.344905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw" event={"ID":"5bea1e32-b5c9-4f76-8bc2-894f24cf3a37","Type":"ContainerDied","Data":"481a963a8cbea3817852c4baadf967684611eece54f60d29abf78316c3b16cbc"}
Sep 10 03:11:37.346746 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.346722    2580 generic.go:356] "Generic (PLEG): container finished" podID="55b16e80-04b5-4743-91e2-5c34e810133a" containerID="78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c" exitCode=0
Sep 10 03:11:37.346852 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.346754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6" event={"ID":"55b16e80-04b5-4743-91e2-5c34e810133a","Type":"ContainerDied","Data":"78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c"}
Sep 10 03:11:37.346852 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.346764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"
Sep 10 03:11:37.346852 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.346788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6" event={"ID":"55b16e80-04b5-4743-91e2-5c34e810133a","Type":"ContainerDied","Data":"7a24457c5d7da37526b580309b65968791969effb0e0d38d365daf91b85bb119"}
Sep 10 03:11:37.354041 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.354016    2580 scope.go:117] "RemoveContainer" containerID="688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838"
Sep 10 03:11:37.354355 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:11:37.354329    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838\": container with ID starting with 688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838 not found: ID does not exist" containerID="688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838"
Sep 10 03:11:37.354454 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.354359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838"} err="failed to get container status \"688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838\": rpc error: code = NotFound desc = could not find container \"688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838\": container with ID starting with 688a89ce5de427b8a5272ddb271689c7ddbcf311cd0511bfee95be028d277838 not found: ID does not exist"
Sep 10 03:11:37.354454 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.354376    2580 scope.go:117] "RemoveContainer" containerID="fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598"
Sep 10 03:11:37.363212 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.363189    2580 scope.go:117] "RemoveContainer" containerID="fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598"
Sep 10 03:11:37.363524 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:11:37.363506    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598\": container with ID starting with fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598 not found: ID does not exist" containerID="fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598"
Sep 10 03:11:37.363587 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.363530    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598"} err="failed to get container status \"fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598\": rpc error: code = NotFound desc = could not find container \"fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598\": container with ID starting with fb7a0bb04c06200ff262043fed46f91112e8e6775352589acdec6f95c0825598 not found: ID does not exist"
Sep 10 03:11:37.363587 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.363549    2580 scope.go:117] "RemoveContainer" containerID="78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c"
Sep 10 03:11:37.370707 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.370684    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"]
Sep 10 03:11:37.372355 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.372334    2580 scope.go:117] "RemoveContainer" containerID="78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c"
Sep 10 03:11:37.372682 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:11:37.372662    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c\": container with ID starting with 78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c not found: ID does not exist" containerID="78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c"
Sep 10 03:11:37.372759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.372691    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c"} err="failed to get container status \"78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c\": rpc error: code = NotFound desc = could not find container \"78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c\": container with ID starting with 78211c27ed7e4922840af6a619feb8e2664ab6d161be6fc24ff273e901210c9c not found: ID does not exist"
Sep 10 03:11:37.374480 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.374461    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/simulator-w0-097e16-kserve-8bf9dc58f-fzs2w"]
Sep 10 03:11:37.386932 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.386903    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"]
Sep 10 03:11:37.393409 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.393384    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-w0-097e16-kserve-57ff44599-vpwrw"]
Sep 10 03:11:37.404866 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.404835    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"]
Sep 10 03:11:37.408869 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:37.408844    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/distinct-2-w0-097e16-kserve-6f78788cb8-dwjv6"]
Sep 10 03:11:39.326474 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:39.326437    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b16e80-04b5-4743-91e2-5c34e810133a" path="/var/lib/kubelet/pods/55b16e80-04b5-4743-91e2-5c34e810133a/volumes"
Sep 10 03:11:39.326881 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:39.326868    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" path="/var/lib/kubelet/pods/5bea1e32-b5c9-4f76-8bc2-894f24cf3a37/volumes"
Sep 10 03:11:39.327272 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:39.327258    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" path="/var/lib/kubelet/pods/cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab/volumes"
Sep 10 03:11:40.094659 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:40.094616    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"]
Sep 10 03:11:40.094934 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:40.094885    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy" containerID="cri-o://20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8" gracePeriod=30
Sep 10 03:11:43.631195 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:43.631134    2580 scope.go:117] "RemoveContainer" containerID="b3cdc2556d3692e16e00ac5fc11739472ad249dac859dfc4fa5cf5ba49920867"
Sep 10 03:11:45.353448 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.353421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 03:11:45.404925 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.404833    2580 generic.go:356] "Generic (PLEG): container finished" podID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerID="20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8" exitCode=0
Sep 10 03:11:45.404925 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.404881    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"
Sep 10 03:11:45.404925 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.404899    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" event={"ID":"96ec6504-208f-4bda-a4d8-2ae1a03203a0","Type":"ContainerDied","Data":"20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8"}
Sep 10 03:11:45.405180 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.404947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv" event={"ID":"96ec6504-208f-4bda-a4d8-2ae1a03203a0","Type":"ContainerDied","Data":"a0a3e7d5684b9232defa934dafd4986abd0358fb701a40b76d19db6e726b64bb"}
Sep 10 03:11:45.405180 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.404972    2580 scope.go:117] "RemoveContainer" containerID="20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8"
Sep 10 03:11:45.414843 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.414813    2580 scope.go:117] "RemoveContainer" containerID="7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15"
Sep 10 03:11:45.424748 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.424724    2580 scope.go:117] "RemoveContainer" containerID="20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8"
Sep 10 03:11:45.425174 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:11:45.425140    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8\": container with ID starting with 20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8 not found: ID does not exist" containerID="20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8"
Sep 10 03:11:45.425255 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.425190    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8"} err="failed to get container status \"20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8\": rpc error: code = NotFound desc = could not find container \"20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8\": container with ID starting with 20e5afb31fa82c8091acbb7f11a78ab594b601aba277f4cdca7509990b6726e8 not found: ID does not exist"
Sep 10 03:11:45.425255 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.425216    2580 scope.go:117] "RemoveContainer" containerID="7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15"
Sep 10 03:11:45.425560 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:11:45.425540    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15\": container with ID starting with 7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15 not found: ID does not exist" containerID="7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15"
Sep 10 03:11:45.425603 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.425568    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15"} err="failed to get container status \"7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15\": rpc error: code = NotFound desc = could not find container \"7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15\": container with ID starting with 7a0324add2e1b10e5eca19cf75b13c473b587b7523d132f9d7a625d3537b5e15 not found: ID does not exist"
Sep 10 03:11:45.446984 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.446942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-token\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.446984 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.446984    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-envoy\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-credential-socket\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447247 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-socket\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j98q\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-kube-api-access-9j98q\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447376 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:45.447376 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-data\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447482 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-certs\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447538 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-podinfo\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447538 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:45.447538 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istiod-ca-cert\") pod \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\" (UID: \"96ec6504-208f-4bda-a4d8-2ae1a03203a0\") "
Sep 10 03:11:45.447698 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:45.447698 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:45.448207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.447910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:11:45.448207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.448089    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.448207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.448110    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.448207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.448124    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.448207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.448137    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.448207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.448151    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.449399 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.449375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:45.449730 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.449710    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:45.449804 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.449724    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-kube-api-access-9j98q" (OuterVolumeSpecName: "kube-api-access-9j98q") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "kube-api-access-9j98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:45.449858 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.449841    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96ec6504-208f-4bda-a4d8-2ae1a03203a0" (UID: "96ec6504-208f-4bda-a4d8-2ae1a03203a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:11:45.548952 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.548912    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.548952 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.548946    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.548952 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.548955    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ec6504-208f-4bda-a4d8-2ae1a03203a0-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.548952 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.548963    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j98q\" (UniqueName: \"kubernetes.io/projected/96ec6504-208f-4bda-a4d8-2ae1a03203a0-kube-api-access-9j98q\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:11:45.728973 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.728929    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"]
Sep 10 03:11:45.732970 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:45.732939    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-097e16-openshift-default-7b675487dd-rvsqv"]
Sep 10 03:11:46.825650 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:46.825615    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:11:47.325583 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:47.325545    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" path="/var/lib/kubelet/pods/96ec6504-208f-4bda-a4d8-2ae1a03203a0/volumes"
Sep 10 03:11:57.005001 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.004965    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"]
Sep 10 03:11:57.005645 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005618    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy"
Sep 10 03:11:57.005645 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005645    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005659    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005665    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005674    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005680    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005687    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b16e80-04b5-4743-91e2-5c34e810133a" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005693    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b16e80-04b5-4743-91e2-5c34e810133a" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005724    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005788    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005795    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ec6504-208f-4bda-a4d8-2ae1a03203a0" containerName="istio-proxy"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005801    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bea1e32-b5c9-4f76-8bc2-894f24cf3a37" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005809    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc48b3c6-fb53-4d4f-80ac-166b8c3b4cab" containerName="main"
Sep 10 03:11:57.005849 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.005816    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b16e80-04b5-4743-91e2-5c34e810133a" containerName="main"
Sep 10 03:11:57.010366 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.010344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.013308 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.013282    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f23388-openshift-default-dockercfg-n6v25\""
Sep 10 03:11:57.020501 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.020062    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"]
Sep 10 03:11:57.170471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0da14b3-ae56-4317-93df-921846e4a06f-istiod-ca-cert\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170679 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0da14b3-ae56-4317-93df-921846e4a06f-istio-podinfo\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170679 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-envoy\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170679 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-data\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170679 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-socket\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170858 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-istio-token\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170858 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-credential-socket\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170858 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-certs\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.170858 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.170789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbpn\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-kube-api-access-8fbpn\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-credential-socket\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-certs\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbpn\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-kube-api-access-8fbpn\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0da14b3-ae56-4317-93df-921846e4a06f-istiod-ca-cert\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0da14b3-ae56-4317-93df-921846e4a06f-istio-podinfo\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-envoy\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-data\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-socket\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.272826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.272769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-istio-token\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.273246 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.273215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-data\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.273336 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.273239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-credential-socket\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.273336 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.273266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-socket\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.273336 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.273295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0da14b3-ae56-4317-93df-921846e4a06f-istiod-ca-cert\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.273459 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.273351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-certs\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.275141 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.275119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-envoy\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.275482 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.275463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0da14b3-ae56-4317-93df-921846e4a06f-istio-podinfo\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.281122 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.281098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-istio-token\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.281570 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.281550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbpn\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-kube-api-access-8fbpn\") pod \"e2e-api-b-f23388-openshift-default-58ff587994-rl257\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") " pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.324323 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.324289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:57.607617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.607581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"]
Sep 10 03:11:57.608614 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:11:57.608577    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0da14b3_ae56_4317_93df_921846e4a06f.slice/crio-6082a46992113fef387620a33a6220a37268ad67e9900f47e8abc11b72ea0d6a WatchSource:0}: Error finding container 6082a46992113fef387620a33a6220a37268ad67e9900f47e8abc11b72ea0d6a: Status 404 returned error can't find the container with id 6082a46992113fef387620a33a6220a37268ad67e9900f47e8abc11b72ea0d6a
Sep 10 03:11:57.611265 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.611231    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:11:57.611392 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.611303    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:11:57.611392 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:57.611330    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:11:58.477334 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:58.477290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" event={"ID":"f0da14b3-ae56-4317-93df-921846e4a06f","Type":"ContainerStarted","Data":"769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b"}
Sep 10 03:11:58.477722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:58.477364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" event={"ID":"f0da14b3-ae56-4317-93df-921846e4a06f","Type":"ContainerStarted","Data":"6082a46992113fef387620a33a6220a37268ad67e9900f47e8abc11b72ea0d6a"}
Sep 10 03:11:58.500093 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:58.500039    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podStartSLOduration=2.5000226 podStartE2EDuration="2.5000226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:11:58.497730596 +0000 UTC m=+1515.647630294" watchObservedRunningTime="2026-09-10 03:11:58.5000226 +0000 UTC m=+1515.649922298"
Sep 10 03:11:59.325315 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:59.325277    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:59.330728 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:59.330700    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:59.482678 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:59.482644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:11:59.483559 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:11:59.483542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:12:02.510011 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.509971    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"]
Sep 10 03:12:02.513937 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.513919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:02.516805 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.516779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f23388\""
Sep 10 03:12:02.517136 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.517118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f23388-dockercfg-9shqj\""
Sep 10 03:12:02.523739 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.523712    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"]
Sep 10 03:12:02.629486 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.629450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a82f55-d3f8-4589-8cd0-772c086fbc65-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:02.629655 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.629528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz668\" (UniqueName: \"kubernetes.io/projected/43a82f55-d3f8-4589-8cd0-772c086fbc65-kube-api-access-rz668\") pod \"payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:02.730704 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.730669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a82f55-d3f8-4589-8cd0-772c086fbc65-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:02.730889 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.730728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz668\" (UniqueName: \"kubernetes.io/projected/43a82f55-d3f8-4589-8cd0-772c086fbc65-kube-api-access-rz668\") pod \"payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:02.731419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.731398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a82f55-d3f8-4589-8cd0-772c086fbc65-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:02.738963 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.738929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz668\" (UniqueName: \"kubernetes.io/projected/43a82f55-d3f8-4589-8cd0-772c086fbc65-kube-api-access-rz668\") pod \"payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:02.826936 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:02.826846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:03.100523 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:03.100494    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"]
Sep 10 03:12:03.102643 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:12:03.102618    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a82f55_d3f8_4589_8cd0_772c086fbc65.slice/crio-640d95f13fd4200487552ba84c617a11d93e208fdd8b6a9685c0fe6147327588 WatchSource:0}: Error finding container 640d95f13fd4200487552ba84c617a11d93e208fdd8b6a9685c0fe6147327588: Status 404 returned error can't find the container with id 640d95f13fd4200487552ba84c617a11d93e208fdd8b6a9685c0fe6147327588
Sep 10 03:12:03.506643 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:03.506538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk" event={"ID":"43a82f55-d3f8-4589-8cd0-772c086fbc65","Type":"ContainerStarted","Data":"9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf"}
Sep 10 03:12:03.506643 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:03.506624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk" event={"ID":"43a82f55-d3f8-4589-8cd0-772c086fbc65","Type":"ContainerStarted","Data":"640d95f13fd4200487552ba84c617a11d93e208fdd8b6a9685c0fe6147327588"}
Sep 10 03:12:03.506882 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:03.506737    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:03.525634 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:03.525584    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk" podStartSLOduration=1.525569814 podStartE2EDuration="1.525569814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:12:03.524788658 +0000 UTC m=+1520.674688357" watchObservedRunningTime="2026-09-10 03:12:03.525569814 +0000 UTC m=+1520.675469512"
Sep 10 03:12:05.798358 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.798321    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"]
Sep 10 03:12:05.802284 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.802263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.805293 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.805271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"e2e-conflict-a-8ec4ab59-kserve-self-signed-certs\""
Sep 10 03:12:05.811568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.811527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"]
Sep 10 03:12:05.864229 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.864157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-dshm\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.864419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.864271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-home\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.864419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.864319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ce752-3bdf-4dbf-960c-59561830beb9-tls-certs\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.864419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.864340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmnz\" (UniqueName: \"kubernetes.io/projected/df2ce752-3bdf-4dbf-960c-59561830beb9-kube-api-access-blmnz\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.864419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.864363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-model-cache\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.864565 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.864446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-tmp-dir\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966151 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-tmp-dir\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966343 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-dshm\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966343 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-home\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966343 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ce752-3bdf-4dbf-960c-59561830beb9-tls-certs\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966512 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmnz\" (UniqueName: \"kubernetes.io/projected/df2ce752-3bdf-4dbf-960c-59561830beb9-kube-api-access-blmnz\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966512 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-model-cache\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966652 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-tmp-dir\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966900 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-model-cache\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.966991 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.966931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-home\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.968690 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.968662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-dshm\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.968973 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.968946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ce752-3bdf-4dbf-960c-59561830beb9-tls-certs\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:05.976024 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:05.975993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmnz\" (UniqueName: \"kubernetes.io/projected/df2ce752-3bdf-4dbf-960c-59561830beb9-kube-api-access-blmnz\") pod \"e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:06.114236 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:06.114192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:06.396499 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:06.396045    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"]
Sep 10 03:12:06.399056 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:12:06.399017    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf2ce752_3bdf_4dbf_960c_59561830beb9.slice/crio-65038ae98ce71ee475d02639a352ff62c64bc0360f88f8b3bba0392ace6a2777 WatchSource:0}: Error finding container 65038ae98ce71ee475d02639a352ff62c64bc0360f88f8b3bba0392ace6a2777: Status 404 returned error can't find the container with id 65038ae98ce71ee475d02639a352ff62c64bc0360f88f8b3bba0392ace6a2777
Sep 10 03:12:06.526802 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:06.526766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr" event={"ID":"df2ce752-3bdf-4dbf-960c-59561830beb9","Type":"ContainerStarted","Data":"4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64"}
Sep 10 03:12:06.526802 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:06.526810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr" event={"ID":"df2ce752-3bdf-4dbf-960c-59561830beb9","Type":"ContainerStarted","Data":"65038ae98ce71ee475d02639a352ff62c64bc0360f88f8b3bba0392ace6a2777"}
Sep 10 03:12:06.545568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:06.545510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr" podStartSLOduration=1.545492114 podStartE2EDuration="1.545492114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:12:06.544668693 +0000 UTC m=+1523.694568404" watchObservedRunningTime="2026-09-10 03:12:06.545492114 +0000 UTC m=+1523.695391888"
Sep 10 03:12:14.515327 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:14.515290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:12:16.115884 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:16.115491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:16.117905 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:16.116448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:16.144239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:16.144198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:16.595422 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:16.595388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:12:23.625758 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:23.625719    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0da14b3-ae56-4317-93df-921846e4a06f" containerID="769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b" exitCode=0
Sep 10 03:12:23.626236 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:23.625790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" event={"ID":"f0da14b3-ae56-4317-93df-921846e4a06f","Type":"ContainerDied","Data":"769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b"}
Sep 10 03:12:23.626301 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:23.626286    2580 scope.go:117] "RemoveContainer" containerID="769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b"
Sep 10 03:12:23.627495 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:23.627469    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:12:23.627578 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:23.627536    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:12:23.627578 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:23.627569    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:12:24.637092 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:24.637055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" event={"ID":"f0da14b3-ae56-4317-93df-921846e4a06f","Type":"ContainerStarted","Data":"566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83"}
Sep 10 03:12:25.327212 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:25.327150    2580 patch_prober.go:28] interesting pod/e2e-api-b-f23388-openshift-default-58ff587994-rl257 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 10 03:12:25.327437 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:25.327252    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 10 03:12:25.331601 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:25.331561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:12:26.325398 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:26.325357    2580 patch_prober.go:28] interesting pod/e2e-api-b-f23388-openshift-default-58ff587994-rl257 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 10 03:12:26.326024 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:26.325977    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 10 03:12:27.325117 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:27.325082    2580 patch_prober.go:28] interesting pod/e2e-api-b-f23388-openshift-default-58ff587994-rl257 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 10 03:12:27.325438 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:27.325406    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 10 03:12:28.256414 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.256375    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"]
Sep 10 03:12:28.259051 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.259033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"
Sep 10 03:12:28.261903 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.261875    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:12:28.262075 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.261918    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:12:28.267569 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.267540    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"]
Sep 10 03:12:28.332189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.328671    2580 patch_prober.go:28] interesting pod/e2e-api-b-f23388-openshift-default-58ff587994-rl257 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 10 03:12:28.332189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.328756    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 10 03:12:28.438007 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.437955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d395aa2-bb47-4dfa-86ed-5f254467886f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0a342-mz4df\" (UID: \"3d395aa2-bb47-4dfa-86ed-5f254467886f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"
Sep 10 03:12:28.539848 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.539760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d395aa2-bb47-4dfa-86ed-5f254467886f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0a342-mz4df\" (UID: \"3d395aa2-bb47-4dfa-86ed-5f254467886f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"
Sep 10 03:12:28.540451 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.540429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d395aa2-bb47-4dfa-86ed-5f254467886f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0a342-mz4df\" (UID: \"3d395aa2-bb47-4dfa-86ed-5f254467886f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"
Sep 10 03:12:28.572117 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.572074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"
Sep 10 03:12:28.867936 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:28.867898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"]
Sep 10 03:12:28.872389 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:12:28.872358    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d395aa2_bb47_4dfa_86ed_5f254467886f.slice/crio-9cb84586c334110a7f2e77cc787ee4b1caafad9c0c476ebfeb430d163d15e57a WatchSource:0}: Error finding container 9cb84586c334110a7f2e77cc787ee4b1caafad9c0c476ebfeb430d163d15e57a: Status 404 returned error can't find the container with id 9cb84586c334110a7f2e77cc787ee4b1caafad9c0c476ebfeb430d163d15e57a
Sep 10 03:12:29.325358 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:29.325316    2580 patch_prober.go:28] interesting pod/e2e-api-b-f23388-openshift-default-58ff587994-rl257 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 10 03:12:29.325594 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:29.325385    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 10 03:12:29.483466 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:29.483373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:12:29.670993 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:29.670945    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df" event={"ID":"3d395aa2-bb47-4dfa-86ed-5f254467886f","Type":"ContainerStarted","Data":"b3f646a0b835ebb0f0e49b82854f79c68a718c6a5e3d53c41a1a9336a6678042"}
Sep 10 03:12:29.671213 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:29.671006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df" event={"ID":"3d395aa2-bb47-4dfa-86ed-5f254467886f","Type":"ContainerStarted","Data":"9cb84586c334110a7f2e77cc787ee4b1caafad9c0c476ebfeb430d163d15e57a"}
Sep 10 03:12:29.687556 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:29.687503    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df" podStartSLOduration=1.687483742 podStartE2EDuration="1.687483742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:12:29.686063269 +0000 UTC m=+1546.835962967" watchObservedRunningTime="2026-09-10 03:12:29.687483742 +0000 UTC m=+1546.837383440"
Sep 10 03:12:30.330228 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:30.330159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:12:30.331352 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:30.331330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:12:30.678747 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:30.678707    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d395aa2-bb47-4dfa-86ed-5f254467886f" containerID="b3f646a0b835ebb0f0e49b82854f79c68a718c6a5e3d53c41a1a9336a6678042" exitCode=0
Sep 10 03:12:30.679278 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:30.678799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df" event={"ID":"3d395aa2-bb47-4dfa-86ed-5f254467886f","Type":"ContainerDied","Data":"b3f646a0b835ebb0f0e49b82854f79c68a718c6a5e3d53c41a1a9336a6678042"}
Sep 10 03:12:31.832830 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:31.832803    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"
Sep 10 03:12:31.881480 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:31.881446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d395aa2-bb47-4dfa-86ed-5f254467886f-maas-api-service-ca\") pod \"3d395aa2-bb47-4dfa-86ed-5f254467886f\" (UID: \"3d395aa2-bb47-4dfa-86ed-5f254467886f\") "
Sep 10 03:12:31.881839 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:31.881816    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d395aa2-bb47-4dfa-86ed-5f254467886f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d395aa2-bb47-4dfa-86ed-5f254467886f" (UID: "3d395aa2-bb47-4dfa-86ed-5f254467886f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:12:31.982978 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:31.982890    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d395aa2-bb47-4dfa-86ed-5f254467886f-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:12:32.022973 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:32.022937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:12:32.691219 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:32.691190    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"
Sep 10 03:12:32.691413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:32.691228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df" event={"ID":"3d395aa2-bb47-4dfa-86ed-5f254467886f","Type":"ContainerDied","Data":"9cb84586c334110a7f2e77cc787ee4b1caafad9c0c476ebfeb430d163d15e57a"}
Sep 10 03:12:32.691413 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:32.691282    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cb84586c334110a7f2e77cc787ee4b1caafad9c0c476ebfeb430d163d15e57a"
Sep 10 03:12:44.237968 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:44.237923    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"]
Sep 10 03:12:44.240291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:44.240266    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0a342-mz4df"]
Sep 10 03:12:45.325839 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:45.325801    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d395aa2-bb47-4dfa-86ed-5f254467886f" path="/var/lib/kubelet/pods/3d395aa2-bb47-4dfa-86ed-5f254467886f/volumes"
Sep 10 03:12:51.179159 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.179124    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"]
Sep 10 03:12:51.179602 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.179589    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d395aa2-bb47-4dfa-86ed-5f254467886f" containerName="revoke-keys"
Sep 10 03:12:51.179647 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.179603    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d395aa2-bb47-4dfa-86ed-5f254467886f" containerName="revoke-keys"
Sep 10 03:12:51.179699 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.179690    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d395aa2-bb47-4dfa-86ed-5f254467886f" containerName="revoke-keys"
Sep 10 03:12:51.182939 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.182921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"
Sep 10 03:12:51.186021 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.185996    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:12:51.186021 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.186015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:12:51.193609 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.193578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"]
Sep 10 03:12:51.297525 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.297487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74e9ef-3380-474b-8490-f2206797f062-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f23388-57lg4\" (UID: \"eb74e9ef-3380-474b-8490-f2206797f062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"
Sep 10 03:12:51.398353 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.398307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74e9ef-3380-474b-8490-f2206797f062-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f23388-57lg4\" (UID: \"eb74e9ef-3380-474b-8490-f2206797f062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"
Sep 10 03:12:51.398996 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.398973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74e9ef-3380-474b-8490-f2206797f062-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f23388-57lg4\" (UID: \"eb74e9ef-3380-474b-8490-f2206797f062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"
Sep 10 03:12:51.495700 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.495603    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"
Sep 10 03:12:51.774005 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.773969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"]
Sep 10 03:12:51.776065 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:12:51.776036    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb74e9ef_3380_474b_8490_f2206797f062.slice/crio-04e26f814ba62ddc23ba227f5dfaa2a18f4a3d6265aa06a2c32ae7f30353caf2 WatchSource:0}: Error finding container 04e26f814ba62ddc23ba227f5dfaa2a18f4a3d6265aa06a2c32ae7f30353caf2: Status 404 returned error can't find the container with id 04e26f814ba62ddc23ba227f5dfaa2a18f4a3d6265aa06a2c32ae7f30353caf2
Sep 10 03:12:51.811694 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:51.811638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4" event={"ID":"eb74e9ef-3380-474b-8490-f2206797f062","Type":"ContainerStarted","Data":"04e26f814ba62ddc23ba227f5dfaa2a18f4a3d6265aa06a2c32ae7f30353caf2"}
Sep 10 03:12:52.818778 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:52.818681    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb74e9ef-3380-474b-8490-f2206797f062" containerID="e8b08542b33b134f6e208ea6112d167a5f6bb4891c841629a566491b1cbaf48b" exitCode=0
Sep 10 03:12:52.819201 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:52.818763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4" event={"ID":"eb74e9ef-3380-474b-8490-f2206797f062","Type":"ContainerDied","Data":"e8b08542b33b134f6e208ea6112d167a5f6bb4891c841629a566491b1cbaf48b"}
Sep 10 03:12:53.968533 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:53.968502    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"
Sep 10 03:12:54.128792 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:54.128737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74e9ef-3380-474b-8490-f2206797f062-maas-api-service-ca\") pod \"eb74e9ef-3380-474b-8490-f2206797f062\" (UID: \"eb74e9ef-3380-474b-8490-f2206797f062\") "
Sep 10 03:12:54.129131 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:54.129105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb74e9ef-3380-474b-8490-f2206797f062-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb74e9ef-3380-474b-8490-f2206797f062" (UID: "eb74e9ef-3380-474b-8490-f2206797f062"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:12:54.230455 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:54.230403    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74e9ef-3380-474b-8490-f2206797f062-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:12:54.832881 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:54.832845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4" event={"ID":"eb74e9ef-3380-474b-8490-f2206797f062","Type":"ContainerDied","Data":"04e26f814ba62ddc23ba227f5dfaa2a18f4a3d6265aa06a2c32ae7f30353caf2"}
Sep 10 03:12:54.832881 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:54.832868    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"
Sep 10 03:12:54.833109 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:54.832900    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04e26f814ba62ddc23ba227f5dfaa2a18f4a3d6265aa06a2c32ae7f30353caf2"
Sep 10 03:12:59.327430 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:12:59.327399    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:13:07.160371 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.160336    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"]
Sep 10 03:13:07.160867 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.160606    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk" podUID="43a82f55-d3f8-4589-8cd0-772c086fbc65" containerName="payload-pre-processing" containerID="cri-o://9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf" gracePeriod=30
Sep 10 03:13:07.388135 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.388089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"]
Sep 10 03:13:07.388409 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.388153    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f23388-57lg4"]
Sep 10 03:13:07.446634 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.446601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:13:07.578377 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.578342    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a82f55-d3f8-4589-8cd0-772c086fbc65-plugins-config-volume\") pod \"43a82f55-d3f8-4589-8cd0-772c086fbc65\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") "
Sep 10 03:13:07.578576 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.578395    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz668\" (UniqueName: \"kubernetes.io/projected/43a82f55-d3f8-4589-8cd0-772c086fbc65-kube-api-access-rz668\") pod \"43a82f55-d3f8-4589-8cd0-772c086fbc65\" (UID: \"43a82f55-d3f8-4589-8cd0-772c086fbc65\") "
Sep 10 03:13:07.578746 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.578722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a82f55-d3f8-4589-8cd0-772c086fbc65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43a82f55-d3f8-4589-8cd0-772c086fbc65" (UID: "43a82f55-d3f8-4589-8cd0-772c086fbc65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:07.580711 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.580685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a82f55-d3f8-4589-8cd0-772c086fbc65-kube-api-access-rz668" (OuterVolumeSpecName: "kube-api-access-rz668") pod "43a82f55-d3f8-4589-8cd0-772c086fbc65" (UID: "43a82f55-d3f8-4589-8cd0-772c086fbc65"). InnerVolumeSpecName "kube-api-access-rz668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:07.679695 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.679600    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a82f55-d3f8-4589-8cd0-772c086fbc65-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.679695 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.679636    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz668\" (UniqueName: \"kubernetes.io/projected/43a82f55-d3f8-4589-8cd0-772c086fbc65-kube-api-access-rz668\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.907421 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.907381    2580 generic.go:356] "Generic (PLEG): container finished" podID="43a82f55-d3f8-4589-8cd0-772c086fbc65" containerID="9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf" exitCode=0
Sep 10 03:13:07.907610 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.907427    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"
Sep 10 03:13:07.907610 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.907474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk" event={"ID":"43a82f55-d3f8-4589-8cd0-772c086fbc65","Type":"ContainerDied","Data":"9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf"}
Sep 10 03:13:07.907610 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.907564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk" event={"ID":"43a82f55-d3f8-4589-8cd0-772c086fbc65","Type":"ContainerDied","Data":"640d95f13fd4200487552ba84c617a11d93e208fdd8b6a9685c0fe6147327588"}
Sep 10 03:13:07.907610 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.907592    2580 scope.go:117] "RemoveContainer" containerID="9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf"
Sep 10 03:13:07.917308 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.917291    2580 scope.go:117] "RemoveContainer" containerID="9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf"
Sep 10 03:13:07.917580 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:13:07.917561    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf\": container with ID starting with 9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf not found: ID does not exist" containerID="9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf"
Sep 10 03:13:07.917638 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.917590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf"} err="failed to get container status \"9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf\": rpc error: code = NotFound desc = could not find container \"9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf\": container with ID starting with 9a409b2c6b67675b2cee1e48d29bce2a4e161570bc431bdcdb7a8165bebbdfbf not found: ID does not exist"
Sep 10 03:13:07.930640 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.930543    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"]
Sep 10 03:13:07.935656 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:07.935614    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f23388-d7cbdc78-9pcsk"]
Sep 10 03:13:09.326614 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:09.326576    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43a82f55-d3f8-4589-8cd0-772c086fbc65" path="/var/lib/kubelet/pods/43a82f55-d3f8-4589-8cd0-772c086fbc65/volumes"
Sep 10 03:13:09.327096 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:09.326963    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb74e9ef-3380-474b-8490-f2206797f062" path="/var/lib/kubelet/pods/eb74e9ef-3380-474b-8490-f2206797f062/volumes"
Sep 10 03:13:10.901863 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:10.901828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"]
Sep 10 03:13:14.150969 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:14.150913    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"]
Sep 10 03:13:14.151453 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:14.151197    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy" containerID="cri-o://566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83" gracePeriod=30
Sep 10 03:13:15.333741 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:15.333704    2580 patch_prober.go:28] interesting pod/e2e-api-b-f23388-openshift-default-58ff587994-rl257 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 03:13:15.334157 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:15.333769    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 03:13:19.425560 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.425482    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:13:19.509342 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-credential-socket\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509545 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0da14b3-ae56-4317-93df-921846e4a06f-istio-podinfo\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509545 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-envoy\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509545 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509436    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fbpn\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-kube-api-access-8fbpn\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509545 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-data\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509755 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-certs\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509755 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:19.509755 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0da14b3-ae56-4317-93df-921846e4a06f-istiod-ca-cert\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509755 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-istio-token\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.509755 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509749    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-socket\") pod \"f0da14b3-ae56-4317-93df-921846e4a06f\" (UID: \"f0da14b3-ae56-4317-93df-921846e4a06f\") "
Sep 10 03:13:19.510002 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.509761    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:19.510067 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.510042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:19.510122 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.510069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0da14b3-ae56-4317-93df-921846e4a06f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:19.510194 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.510139    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.510194 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.510145    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:19.510307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.510177    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.510307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.510219    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.511900 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.511873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:19.512044 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.511907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-kube-api-access-8fbpn" (OuterVolumeSpecName: "kube-api-access-8fbpn") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "kube-api-access-8fbpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:19.512141 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.512114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:19.512309 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.512287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0da14b3-ae56-4317-93df-921846e4a06f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0da14b3-ae56-4317-93df-921846e4a06f" (UID: "f0da14b3-ae56-4317-93df-921846e4a06f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:13:19.611543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.611487    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0da14b3-ae56-4317-93df-921846e4a06f-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.611543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.611537    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.611543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.611546    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.611543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.611555    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0da14b3-ae56-4317-93df-921846e4a06f-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.611543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.611564    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0da14b3-ae56-4317-93df-921846e4a06f-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.611876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.611572    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbpn\" (UniqueName: \"kubernetes.io/projected/f0da14b3-ae56-4317-93df-921846e4a06f-kube-api-access-8fbpn\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:19.980181 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.980136    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0da14b3-ae56-4317-93df-921846e4a06f" containerID="566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83" exitCode=0
Sep 10 03:13:19.980367 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.980197    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"
Sep 10 03:13:19.980367 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.980221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" event={"ID":"f0da14b3-ae56-4317-93df-921846e4a06f","Type":"ContainerDied","Data":"566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83"}
Sep 10 03:13:19.980367 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.980274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257" event={"ID":"f0da14b3-ae56-4317-93df-921846e4a06f","Type":"ContainerDied","Data":"6082a46992113fef387620a33a6220a37268ad67e9900f47e8abc11b72ea0d6a"}
Sep 10 03:13:19.980367 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.980297    2580 scope.go:117] "RemoveContainer" containerID="566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83"
Sep 10 03:13:19.993259 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:19.993232    2580 scope.go:117] "RemoveContainer" containerID="769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b"
Sep 10 03:13:20.003431 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:20.003406    2580 scope.go:117] "RemoveContainer" containerID="566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83"
Sep 10 03:13:20.003835 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:13:20.003813    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83\": container with ID starting with 566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83 not found: ID does not exist" containerID="566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83"
Sep 10 03:13:20.003890 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:20.003846    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83"} err="failed to get container status \"566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83\": rpc error: code = NotFound desc = could not find container \"566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83\": container with ID starting with 566f5cce95aef81fea24881870aa39916791f38687c60a06ec1df57d2ef3bb83 not found: ID does not exist"
Sep 10 03:13:20.003890 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:20.003867    2580 scope.go:117] "RemoveContainer" containerID="769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b"
Sep 10 03:13:20.004264 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:13:20.004195    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b\": container with ID starting with 769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b not found: ID does not exist" containerID="769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b"
Sep 10 03:13:20.004264 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:20.004228    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b"} err="failed to get container status \"769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b\": rpc error: code = NotFound desc = could not find container \"769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b\": container with ID starting with 769a88a4b4f99470a75bbddd626d93d9a93ec7dbf642625c23dd83353327a45b not found: ID does not exist"
Sep 10 03:13:20.005618 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:20.005591    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"]
Sep 10 03:13:20.009450 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:20.009427    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f23388-openshift-default-58ff587994-rl257"]
Sep 10 03:13:21.327018 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:21.326979    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" path="/var/lib/kubelet/pods/f0da14b3-ae56-4317-93df-921846e4a06f/volumes"
Sep 10 03:13:25.923812 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:25.923765    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr" podUID="df2ce752-3bdf-4dbf-960c-59561830beb9" containerName="main" containerID="cri-o://4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64" gracePeriod=45
Sep 10 03:13:26.181053 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.180979    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:13:26.281120 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281070    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-home\") pod \"df2ce752-3bdf-4dbf-960c-59561830beb9\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") "
Sep 10 03:13:26.281364 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blmnz\" (UniqueName: \"kubernetes.io/projected/df2ce752-3bdf-4dbf-960c-59561830beb9-kube-api-access-blmnz\") pod \"df2ce752-3bdf-4dbf-960c-59561830beb9\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") "
Sep 10 03:13:26.281364 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-model-cache\") pod \"df2ce752-3bdf-4dbf-960c-59561830beb9\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") "
Sep 10 03:13:26.281364 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-dshm\") pod \"df2ce752-3bdf-4dbf-960c-59561830beb9\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") "
Sep 10 03:13:26.281364 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-tmp-dir\") pod \"df2ce752-3bdf-4dbf-960c-59561830beb9\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") "
Sep 10 03:13:26.281364 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ce752-3bdf-4dbf-960c-59561830beb9-tls-certs\") pod \"df2ce752-3bdf-4dbf-960c-59561830beb9\" (UID: \"df2ce752-3bdf-4dbf-960c-59561830beb9\") "
Sep 10 03:13:26.281652 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281413    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-home" (OuterVolumeSpecName: "home") pod "df2ce752-3bdf-4dbf-960c-59561830beb9" (UID: "df2ce752-3bdf-4dbf-960c-59561830beb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:26.281652 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "df2ce752-3bdf-4dbf-960c-59561830beb9" (UID: "df2ce752-3bdf-4dbf-960c-59561830beb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:26.281652 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df2ce752-3bdf-4dbf-960c-59561830beb9" (UID: "df2ce752-3bdf-4dbf-960c-59561830beb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:26.281810 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281760    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:26.281810 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281773    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:26.281810 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.281782    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:26.283578 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.283546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-dshm" (OuterVolumeSpecName: "dshm") pod "df2ce752-3bdf-4dbf-960c-59561830beb9" (UID: "df2ce752-3bdf-4dbf-960c-59561830beb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:26.283578 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.283558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2ce752-3bdf-4dbf-960c-59561830beb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df2ce752-3bdf-4dbf-960c-59561830beb9" (UID: "df2ce752-3bdf-4dbf-960c-59561830beb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:13:26.283722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.283631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2ce752-3bdf-4dbf-960c-59561830beb9-kube-api-access-blmnz" (OuterVolumeSpecName: "kube-api-access-blmnz") pod "df2ce752-3bdf-4dbf-960c-59561830beb9" (UID: "df2ce752-3bdf-4dbf-960c-59561830beb9"). InnerVolumeSpecName "kube-api-access-blmnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:26.383549 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.383513    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmnz\" (UniqueName: \"kubernetes.io/projected/df2ce752-3bdf-4dbf-960c-59561830beb9-kube-api-access-blmnz\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:26.383549 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.383541    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ce752-3bdf-4dbf-960c-59561830beb9-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:26.383549 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:26.383552    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ce752-3bdf-4dbf-960c-59561830beb9-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:27.024499 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.024462    2580 generic.go:356] "Generic (PLEG): container finished" podID="df2ce752-3bdf-4dbf-960c-59561830beb9" containerID="4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64" exitCode=0
Sep 10 03:13:27.025000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.024507    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"
Sep 10 03:13:27.025000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.024519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr" event={"ID":"df2ce752-3bdf-4dbf-960c-59561830beb9","Type":"ContainerDied","Data":"4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64"}
Sep 10 03:13:27.025000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.024553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr" event={"ID":"df2ce752-3bdf-4dbf-960c-59561830beb9","Type":"ContainerDied","Data":"65038ae98ce71ee475d02639a352ff62c64bc0360f88f8b3bba0392ace6a2777"}
Sep 10 03:13:27.025000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.024572    2580 scope.go:117] "RemoveContainer" containerID="4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64"
Sep 10 03:13:27.034665 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.034645    2580 scope.go:117] "RemoveContainer" containerID="4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64"
Sep 10 03:13:27.035059 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:13:27.035035    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64\": container with ID starting with 4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64 not found: ID does not exist" containerID="4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64"
Sep 10 03:13:27.035109 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.035068    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64"} err="failed to get container status \"4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64\": rpc error: code = NotFound desc = could not find container \"4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64\": container with ID starting with 4a78a60f654e227b1570d20f13408f8717f7ccb444a47a367ee4a8c685b75e64 not found: ID does not exist"
Sep 10 03:13:27.051522 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.051485    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"]
Sep 10 03:13:27.056459 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.056430    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-a-8ec4ab59-kserve-64f6b5bb6-z2ftr"]
Sep 10 03:13:27.325452 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:27.325358    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2ce752-3bdf-4dbf-960c-59561830beb9" path="/var/lib/kubelet/pods/df2ce752-3bdf-4dbf-960c-59561830beb9/volumes"
Sep 10 03:13:38.314587 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:38.314544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:13:43.182535 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.182493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"]
Sep 10 03:13:43.183031 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.182800    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592" podUID="210bd681-22ec-44fb-aef3-ac977f5c9fc0" containerName="payload-processing" containerID="cri-o://7dfd91109781b091b8c16e8bf6d63789253afc27e6bfa16e3152d5ea3161f0e6" gracePeriod=30
Sep 10 03:13:43.192401 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.192371    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"]
Sep 10 03:13:43.192623 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.192595    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs" podUID="31c2467b-86d5-4c28-ab85-44838bddef71" containerName="payload-pre-processing" containerID="cri-o://c094912d5bf9604bf307235830de4e951d3d52ed43688ab60ac3419199290830" gracePeriod=30
Sep 10 03:13:43.452314 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.452281    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:13:43.491268 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.491233    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:13:43.564018 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.563982    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/210bd681-22ec-44fb-aef3-ac977f5c9fc0-plugins-config-volume\") pod \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") "
Sep 10 03:13:43.564238 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.564039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg229\" (UniqueName: \"kubernetes.io/projected/210bd681-22ec-44fb-aef3-ac977f5c9fc0-kube-api-access-sg229\") pod \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\" (UID: \"210bd681-22ec-44fb-aef3-ac977f5c9fc0\") "
Sep 10 03:13:43.564238 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.564119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8czsl\" (UniqueName: \"kubernetes.io/projected/31c2467b-86d5-4c28-ab85-44838bddef71-kube-api-access-8czsl\") pod \"31c2467b-86d5-4c28-ab85-44838bddef71\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") "
Sep 10 03:13:43.564238 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.564204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c2467b-86d5-4c28-ab85-44838bddef71-plugins-config-volume\") pod \"31c2467b-86d5-4c28-ab85-44838bddef71\" (UID: \"31c2467b-86d5-4c28-ab85-44838bddef71\") "
Sep 10 03:13:43.564576 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.564541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210bd681-22ec-44fb-aef3-ac977f5c9fc0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "210bd681-22ec-44fb-aef3-ac977f5c9fc0" (UID: "210bd681-22ec-44fb-aef3-ac977f5c9fc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:43.564662 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.564623    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c2467b-86d5-4c28-ab85-44838bddef71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31c2467b-86d5-4c28-ab85-44838bddef71" (UID: "31c2467b-86d5-4c28-ab85-44838bddef71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:43.564723 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.564682    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/210bd681-22ec-44fb-aef3-ac977f5c9fc0-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:43.566432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.566405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210bd681-22ec-44fb-aef3-ac977f5c9fc0-kube-api-access-sg229" (OuterVolumeSpecName: "kube-api-access-sg229") pod "210bd681-22ec-44fb-aef3-ac977f5c9fc0" (UID: "210bd681-22ec-44fb-aef3-ac977f5c9fc0"). InnerVolumeSpecName "kube-api-access-sg229". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:43.566543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.566525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c2467b-86d5-4c28-ab85-44838bddef71-kube-api-access-8czsl" (OuterVolumeSpecName: "kube-api-access-8czsl") pod "31c2467b-86d5-4c28-ab85-44838bddef71" (UID: "31c2467b-86d5-4c28-ab85-44838bddef71"). InnerVolumeSpecName "kube-api-access-8czsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:43.666113 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.666071    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg229\" (UniqueName: \"kubernetes.io/projected/210bd681-22ec-44fb-aef3-ac977f5c9fc0-kube-api-access-sg229\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:43.666113 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.666106    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8czsl\" (UniqueName: \"kubernetes.io/projected/31c2467b-86d5-4c28-ab85-44838bddef71-kube-api-access-8czsl\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:43.666113 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.666117    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c2467b-86d5-4c28-ab85-44838bddef71-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:13:43.680801 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.680768    2580 scope.go:117] "RemoveContainer" containerID="7dfd91109781b091b8c16e8bf6d63789253afc27e6bfa16e3152d5ea3161f0e6"
Sep 10 03:13:43.692196 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:43.692144    2580 scope.go:117] "RemoveContainer" containerID="c094912d5bf9604bf307235830de4e951d3d52ed43688ab60ac3419199290830"
Sep 10 03:13:44.122000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.121956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs" event={"ID":"31c2467b-86d5-4c28-ab85-44838bddef71","Type":"ContainerDied","Data":"c094912d5bf9604bf307235830de4e951d3d52ed43688ab60ac3419199290830"}
Sep 10 03:13:44.122229 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.122017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"
Sep 10 03:13:44.122229 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.122039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs" event={"ID":"31c2467b-86d5-4c28-ab85-44838bddef71","Type":"ContainerDied","Data":"64e2cf5621faf93e00cfaa151026928cf58f5f0bd92e3352b17ed6f08d74884e"}
Sep 10 03:13:44.122229 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.122065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592" event={"ID":"210bd681-22ec-44fb-aef3-ac977f5c9fc0","Type":"ContainerDied","Data":"7dfd91109781b091b8c16e8bf6d63789253afc27e6bfa16e3152d5ea3161f0e6"}
Sep 10 03:13:44.122229 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.122092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592" event={"ID":"210bd681-22ec-44fb-aef3-ac977f5c9fc0","Type":"ContainerDied","Data":"f593f39177e4a1e2e18661627d704bbe2a2e1b76cdbb6a1b324a6289af3e9356"}
Sep 10 03:13:44.122229 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.122020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"
Sep 10 03:13:44.189879 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.189842    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"]
Sep 10 03:13:44.194290 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.194258    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3058d-6fd8b65645-6b592"]
Sep 10 03:13:44.209338 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.209305    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"]
Sep 10 03:13:44.217885 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:44.217850    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3058d-7d6bb5b54-5gnfs"]
Sep 10 03:13:45.325513 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:45.325476    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="210bd681-22ec-44fb-aef3-ac977f5c9fc0" path="/var/lib/kubelet/pods/210bd681-22ec-44fb-aef3-ac977f5c9fc0/volumes"
Sep 10 03:13:45.326044 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:13:45.325987    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c2467b-86d5-4c28-ab85-44838bddef71" path="/var/lib/kubelet/pods/31c2467b-86d5-4c28-ab85-44838bddef71/volumes"
Sep 10 03:14:11.533852 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:11.533812    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:14:30.613778 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:30.613733    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"]
Sep 10 03:14:36.415757 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:36.415720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:14:45.634418 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:45.634368    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k" podUID="4f9fedd4-b928-40a6-853f-858b180cc714" containerName="main" containerID="cri-o://09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909" gracePeriod=45
Sep 10 03:14:45.893187 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:45.893093    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:14:46.005922 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.005881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z55h8\" (UniqueName: \"kubernetes.io/projected/4f9fedd4-b928-40a6-853f-858b180cc714-kube-api-access-z55h8\") pod \"4f9fedd4-b928-40a6-853f-858b180cc714\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") "
Sep 10 03:14:46.006118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.005951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9fedd4-b928-40a6-853f-858b180cc714-tls-certs\") pod \"4f9fedd4-b928-40a6-853f-858b180cc714\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") "
Sep 10 03:14:46.006118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.005992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-tmp-dir\") pod \"4f9fedd4-b928-40a6-853f-858b180cc714\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") "
Sep 10 03:14:46.006118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-dshm\") pod \"4f9fedd4-b928-40a6-853f-858b180cc714\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") "
Sep 10 03:14:46.006324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-model-cache\") pod \"4f9fedd4-b928-40a6-853f-858b180cc714\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") "
Sep 10 03:14:46.006324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006276    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-home\") pod \"4f9fedd4-b928-40a6-853f-858b180cc714\" (UID: \"4f9fedd4-b928-40a6-853f-858b180cc714\") "
Sep 10 03:14:46.006439 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006308    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f9fedd4-b928-40a6-853f-858b180cc714" (UID: "4f9fedd4-b928-40a6-853f-858b180cc714"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:14:46.006497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f9fedd4-b928-40a6-853f-858b180cc714" (UID: "4f9fedd4-b928-40a6-853f-858b180cc714"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:14:46.006543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006507    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-home" (OuterVolumeSpecName: "home") pod "4f9fedd4-b928-40a6-853f-858b180cc714" (UID: "4f9fedd4-b928-40a6-853f-858b180cc714"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:14:46.006708 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006693    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:14:46.006765 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006709    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:14:46.006765 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.006718    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:14:46.008387 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.008361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-dshm" (OuterVolumeSpecName: "dshm") pod "4f9fedd4-b928-40a6-853f-858b180cc714" (UID: "4f9fedd4-b928-40a6-853f-858b180cc714"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:14:46.008700 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.008678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9fedd4-b928-40a6-853f-858b180cc714-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f9fedd4-b928-40a6-853f-858b180cc714" (UID: "4f9fedd4-b928-40a6-853f-858b180cc714"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:14:46.008751 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.008694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9fedd4-b928-40a6-853f-858b180cc714-kube-api-access-z55h8" (OuterVolumeSpecName: "kube-api-access-z55h8") pod "4f9fedd4-b928-40a6-853f-858b180cc714" (UID: "4f9fedd4-b928-40a6-853f-858b180cc714"). InnerVolumeSpecName "kube-api-access-z55h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:14:46.107491 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.107455    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9fedd4-b928-40a6-853f-858b180cc714-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:14:46.107491 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.107487    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z55h8\" (UniqueName: \"kubernetes.io/projected/4f9fedd4-b928-40a6-853f-858b180cc714-kube-api-access-z55h8\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:14:46.107491 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.107497    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9fedd4-b928-40a6-853f-858b180cc714-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:14:46.464755 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.464717    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f9fedd4-b928-40a6-853f-858b180cc714" containerID="09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909" exitCode=0
Sep 10 03:14:46.464940 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.464774    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"
Sep 10 03:14:46.464940 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.464789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k" event={"ID":"4f9fedd4-b928-40a6-853f-858b180cc714","Type":"ContainerDied","Data":"09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909"}
Sep 10 03:14:46.464940 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.464828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k" event={"ID":"4f9fedd4-b928-40a6-853f-858b180cc714","Type":"ContainerDied","Data":"c2b0e8e92a4bfdb3cf21b690e35ac0d660d25c196c4b9d9b365e7e0b748165ee"}
Sep 10 03:14:46.464940 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.464848    2580 scope.go:117] "RemoveContainer" containerID="09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909"
Sep 10 03:14:46.474406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.474385    2580 scope.go:117] "RemoveContainer" containerID="09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909"
Sep 10 03:14:46.474721 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:14:46.474702    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909\": container with ID starting with 09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909 not found: ID does not exist" containerID="09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909"
Sep 10 03:14:46.474776 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.474732    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909"} err="failed to get container status \"09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909\": rpc error: code = NotFound desc = could not find container \"09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909\": container with ID starting with 09fb2b830f66efef6e505c0d352ecfc62c56b6ca7f54505fa3c1fe533a1d1909 not found: ID does not exist"
Sep 10 03:14:46.490605 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.490564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"]
Sep 10 03:14:46.497552 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:46.497518    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/auth-test-model-af7727-kserve-58b759fbc5-rxf5k"]
Sep 10 03:14:47.325379 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:14:47.325344    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f9fedd4-b928-40a6-853f-858b180cc714" path="/var/lib/kubelet/pods/4f9fedd4-b928-40a6-853f-858b180cc714/volumes"
Sep 10 03:15:00.141675 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.141631    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn"]
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142093    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142107    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2ce752-3bdf-4dbf-960c-59561830beb9" containerName="main"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142124    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2ce752-3bdf-4dbf-960c-59561830beb9" containerName="main"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142132    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210bd681-22ec-44fb-aef3-ac977f5c9fc0" containerName="payload-processing"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142139    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="210bd681-22ec-44fb-aef3-ac977f5c9fc0" containerName="payload-processing"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142150    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb74e9ef-3380-474b-8490-f2206797f062" containerName="revoke-keys"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142155    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb74e9ef-3380-474b-8490-f2206797f062" containerName="revoke-keys"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142179    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142185    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142195    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a82f55-d3f8-4589-8cd0-772c086fbc65" containerName="payload-pre-processing"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142201    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a82f55-d3f8-4589-8cd0-772c086fbc65" containerName="payload-pre-processing"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142212    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c2467b-86d5-4c28-ab85-44838bddef71" containerName="payload-pre-processing"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c2467b-86d5-4c28-ab85-44838bddef71" containerName="payload-pre-processing"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142223    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f9fedd4-b928-40a6-853f-858b180cc714" containerName="main"
Sep 10 03:15:00.142239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142228    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f9fedd4-b928-40a6-853f-858b180cc714" containerName="main"
Sep 10 03:15:00.142837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142290    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="210bd681-22ec-44fb-aef3-ac977f5c9fc0" containerName="payload-processing"
Sep 10 03:15:00.142837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142299    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2ce752-3bdf-4dbf-960c-59561830beb9" containerName="main"
Sep 10 03:15:00.142837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142307    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy"
Sep 10 03:15:00.142837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142312    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f9fedd4-b928-40a6-853f-858b180cc714" containerName="main"
Sep 10 03:15:00.142837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142317    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c2467b-86d5-4c28-ab85-44838bddef71" containerName="payload-pre-processing"
Sep 10 03:15:00.142837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142324    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="43a82f55-d3f8-4589-8cd0-772c086fbc65" containerName="payload-pre-processing"
Sep 10 03:15:00.142837 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.142330    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb74e9ef-3380-474b-8490-f2206797f062" containerName="revoke-keys"
Sep 10 03:15:00.146784 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.146763    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"]
Sep 10 03:15:00.146907 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.146887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn"
Sep 10 03:15:00.147347 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.147328    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0da14b3-ae56-4317-93df-921846e4a06f" containerName="istio-proxy"
Sep 10 03:15:00.149733 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.149707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:15:00.150431 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.150409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"
Sep 10 03:15:00.156282 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.156260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn"]
Sep 10 03:15:00.159622 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.159591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"]
Sep 10 03:15:00.163118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.163094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn"
Sep 10 03:15:00.170658 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.170636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"
Sep 10 03:15:00.375011 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.374976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:15:00.467517 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.467320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"]
Sep 10 03:15:00.469991 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:15:00.469950    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3e5a46_c6db_4ff0_ada8_e39a25644a6b.slice/crio-0dc1c56ce950fdd0a1b7211d8bd98d7798142e707619126c217ed9a992df1662 WatchSource:0}: Error finding container 0dc1c56ce950fdd0a1b7211d8bd98d7798142e707619126c217ed9a992df1662: Status 404 returned error can't find the container with id 0dc1c56ce950fdd0a1b7211d8bd98d7798142e707619126c217ed9a992df1662
Sep 10 03:15:00.470539 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.470516    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn"]
Sep 10 03:15:00.473008 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:15:00.472985    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446f3df1_0d84_42a1_b5e0_3158ec89c6f8.slice/crio-a7538d12e6304d17ea401f031540dd079f15f0912ce20b0ffb1a4467e73e1be2 WatchSource:0}: Error finding container a7538d12e6304d17ea401f031540dd079f15f0912ce20b0ffb1a4467e73e1be2: Status 404 returned error can't find the container with id a7538d12e6304d17ea401f031540dd079f15f0912ce20b0ffb1a4467e73e1be2
Sep 10 03:15:00.542061 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.542013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d" event={"ID":"de3e5a46-c6db-4ff0-ada8-e39a25644a6b","Type":"ContainerStarted","Data":"0dc1c56ce950fdd0a1b7211d8bd98d7798142e707619126c217ed9a992df1662"}
Sep 10 03:15:00.543284 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:00.543255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn" event={"ID":"446f3df1-0d84-42a1-b5e0-3158ec89c6f8","Type":"ContainerStarted","Data":"a7538d12e6304d17ea401f031540dd079f15f0912ce20b0ffb1a4467e73e1be2"}
Sep 10 03:15:01.550909 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:01.550868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d" event={"ID":"de3e5a46-c6db-4ff0-ada8-e39a25644a6b","Type":"ContainerStarted","Data":"9a05395d5c4b08ed19772a1408ccb00e3e83fce53db8eeef1bc6285f19b234c7"}
Sep 10 03:15:01.552641 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:01.552607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn" event={"ID":"446f3df1-0d84-42a1-b5e0-3158ec89c6f8","Type":"ContainerStarted","Data":"f86b3c92f4e825b66a0e3f40ad165d467925605f6fc7311c7cb07857d1bff9f9"}
Sep 10 03:15:01.572946 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:01.572896    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d" podStartSLOduration=1.57287829 podStartE2EDuration="1.57287829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:15:01.571766436 +0000 UTC m=+1698.721666136" watchObservedRunningTime="2026-09-10 03:15:01.57287829 +0000 UTC m=+1698.722777987"
Sep 10 03:15:01.593381 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:01.593317    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn" podStartSLOduration=1.59329378 podStartE2EDuration="1.59329378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:15:01.58987418 +0000 UTC m=+1698.739773879" watchObservedRunningTime="2026-09-10 03:15:01.59329378 +0000 UTC m=+1698.743193490"
Sep 10 03:15:02.566579 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:02.566535    2580 generic.go:356] "Generic (PLEG): container finished" podID="446f3df1-0d84-42a1-b5e0-3158ec89c6f8" containerID="f86b3c92f4e825b66a0e3f40ad165d467925605f6fc7311c7cb07857d1bff9f9" exitCode=0
Sep 10 03:15:02.566962 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:02.566623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn" event={"ID":"446f3df1-0d84-42a1-b5e0-3158ec89c6f8","Type":"ContainerDied","Data":"f86b3c92f4e825b66a0e3f40ad165d467925605f6fc7311c7cb07857d1bff9f9"}
Sep 10 03:15:02.568490 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:02.568466    2580 generic.go:356] "Generic (PLEG): container finished" podID="de3e5a46-c6db-4ff0-ada8-e39a25644a6b" containerID="9a05395d5c4b08ed19772a1408ccb00e3e83fce53db8eeef1bc6285f19b234c7" exitCode=0
Sep 10 03:15:02.568606 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:02.568501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d" event={"ID":"de3e5a46-c6db-4ff0-ada8-e39a25644a6b","Type":"ContainerDied","Data":"9a05395d5c4b08ed19772a1408ccb00e3e83fce53db8eeef1bc6285f19b234c7"}
Sep 10 03:15:03.722541 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:03.722510    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"
Sep 10 03:15:03.725638 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:03.725620    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn"
Sep 10 03:15:04.581367 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:04.581337    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"
Sep 10 03:15:04.581562 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:04.581335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d" event={"ID":"de3e5a46-c6db-4ff0-ada8-e39a25644a6b","Type":"ContainerDied","Data":"0dc1c56ce950fdd0a1b7211d8bd98d7798142e707619126c217ed9a992df1662"}
Sep 10 03:15:04.581562 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:04.581456    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc1c56ce950fdd0a1b7211d8bd98d7798142e707619126c217ed9a992df1662"
Sep 10 03:15:04.582663 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:04.582639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn" event={"ID":"446f3df1-0d84-42a1-b5e0-3158ec89c6f8","Type":"ContainerDied","Data":"a7538d12e6304d17ea401f031540dd079f15f0912ce20b0ffb1a4467e73e1be2"}
Sep 10 03:15:04.582749 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:04.582675    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7538d12e6304d17ea401f031540dd079f15f0912ce20b0ffb1a4467e73e1be2"
Sep 10 03:15:04.582749 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:04.582630    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816835-xphvn"
Sep 10 03:15:22.911045 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:22.911010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:15:26.568321 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:26.568281    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"]
Sep 10 03:15:26.586407 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:26.586369    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"]
Sep 10 03:15:26.605459 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:26.605425    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"]
Sep 10 03:15:41.589458 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.589408    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q" podUID="ee2514ee-183d-49f4-9d34-d7bcbab49933" containerName="main" containerID="cri-o://ed552828165af892f79ad119c1d989cf23bb6df6f9b770ab58582da96377bf09" gracePeriod=45
Sep 10 03:15:41.606811 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.606752    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449" podUID="69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" containerName="main" containerID="cri-o://e79371ca404e7283aff3d40a4a96fb8624e4e09bd24e9e5f298ebb95ca618f25" gracePeriod=45
Sep 10 03:15:41.625874 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.625818    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc" podUID="40ef286d-ed76-4d2f-9457-6257d364a2ed" containerName="main" containerID="cri-o://f89905ac625505062f130f75beb7544072bf2de34577f163af0edb169c141d72" gracePeriod=45
Sep 10 03:15:41.791538 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.791346    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee2514ee-183d-49f4-9d34-d7bcbab49933" containerID="ed552828165af892f79ad119c1d989cf23bb6df6f9b770ab58582da96377bf09" exitCode=0
Sep 10 03:15:41.791538 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.791482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q" event={"ID":"ee2514ee-183d-49f4-9d34-d7bcbab49933","Type":"ContainerDied","Data":"ed552828165af892f79ad119c1d989cf23bb6df6f9b770ab58582da96377bf09"}
Sep 10 03:15:41.794525 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.794494    2580 generic.go:356] "Generic (PLEG): container finished" podID="40ef286d-ed76-4d2f-9457-6257d364a2ed" containerID="f89905ac625505062f130f75beb7544072bf2de34577f163af0edb169c141d72" exitCode=0
Sep 10 03:15:41.794689 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.794621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc" event={"ID":"40ef286d-ed76-4d2f-9457-6257d364a2ed","Type":"ContainerDied","Data":"f89905ac625505062f130f75beb7544072bf2de34577f163af0edb169c141d72"}
Sep 10 03:15:41.797154 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.797118    2580 generic.go:356] "Generic (PLEG): container finished" podID="69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" containerID="e79371ca404e7283aff3d40a4a96fb8624e4e09bd24e9e5f298ebb95ca618f25" exitCode=0
Sep 10 03:15:41.797334 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.797203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449" event={"ID":"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00","Type":"ContainerDied","Data":"e79371ca404e7283aff3d40a4a96fb8624e4e09bd24e9e5f298ebb95ca618f25"}
Sep 10 03:15:41.900223 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.900194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:15:41.930556 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.930519    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:15:41.935025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.934992    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:15:41.995561 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.995528    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2514ee-183d-49f4-9d34-d7bcbab49933-tls-certs\") pod \"ee2514ee-183d-49f4-9d34-d7bcbab49933\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") "
Sep 10 03:15:41.995722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.995641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-dshm\") pod \"ee2514ee-183d-49f4-9d34-d7bcbab49933\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") "
Sep 10 03:15:41.995722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.995663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cslnt\" (UniqueName: \"kubernetes.io/projected/ee2514ee-183d-49f4-9d34-d7bcbab49933-kube-api-access-cslnt\") pod \"ee2514ee-183d-49f4-9d34-d7bcbab49933\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") "
Sep 10 03:15:41.995722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.995694    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-model-cache\") pod \"ee2514ee-183d-49f4-9d34-d7bcbab49933\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") "
Sep 10 03:15:41.995722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.995717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-home\") pod \"ee2514ee-183d-49f4-9d34-d7bcbab49933\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") "
Sep 10 03:15:41.996009 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.995740    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-tmp-dir\") pod \"ee2514ee-183d-49f4-9d34-d7bcbab49933\" (UID: \"ee2514ee-183d-49f4-9d34-d7bcbab49933\") "
Sep 10 03:15:41.996068 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.995977    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee2514ee-183d-49f4-9d34-d7bcbab49933" (UID: "ee2514ee-183d-49f4-9d34-d7bcbab49933"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.996068 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.996015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-home" (OuterVolumeSpecName: "home") pod "ee2514ee-183d-49f4-9d34-d7bcbab49933" (UID: "ee2514ee-183d-49f4-9d34-d7bcbab49933"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.996249 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.996220    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee2514ee-183d-49f4-9d34-d7bcbab49933" (UID: "ee2514ee-183d-49f4-9d34-d7bcbab49933"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.997796 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.997771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee2514ee-183d-49f4-9d34-d7bcbab49933-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee2514ee-183d-49f4-9d34-d7bcbab49933" (UID: "ee2514ee-183d-49f4-9d34-d7bcbab49933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:15:41.998083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.998053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-dshm" (OuterVolumeSpecName: "dshm") pod "ee2514ee-183d-49f4-9d34-d7bcbab49933" (UID: "ee2514ee-183d-49f4-9d34-d7bcbab49933"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.998083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:41.998072    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2514ee-183d-49f4-9d34-d7bcbab49933-kube-api-access-cslnt" (OuterVolumeSpecName: "kube-api-access-cslnt") pod "ee2514ee-183d-49f4-9d34-d7bcbab49933" (UID: "ee2514ee-183d-49f4-9d34-d7bcbab49933"). InnerVolumeSpecName "kube-api-access-cslnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:42.096759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.096705    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef286d-ed76-4d2f-9457-6257d364a2ed-tls-certs\") pod \"40ef286d-ed76-4d2f-9457-6257d364a2ed\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") "
Sep 10 03:15:42.096976 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.096789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-dshm\") pod \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") "
Sep 10 03:15:42.096976 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.096809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-model-cache\") pod \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") "
Sep 10 03:15:42.096976 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.096838    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-tmp-dir\") pod \"40ef286d-ed76-4d2f-9457-6257d364a2ed\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") "
Sep 10 03:15:42.096976 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.096881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-model-cache\") pod \"40ef286d-ed76-4d2f-9457-6257d364a2ed\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") "
Sep 10 03:15:42.096976 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.096906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tls-certs\") pod \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") "
Sep 10 03:15:42.096976 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.096944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tmp-dir\") pod \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") "
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097000    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-dshm\") pod \"40ef286d-ed76-4d2f-9457-6257d364a2ed\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") "
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097047    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-home\") pod \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") "
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-home\") pod \"40ef286d-ed76-4d2f-9457-6257d364a2ed\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") "
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqz5x\" (UniqueName: \"kubernetes.io/projected/40ef286d-ed76-4d2f-9457-6257d364a2ed-kube-api-access-wqz5x\") pod \"40ef286d-ed76-4d2f-9457-6257d364a2ed\" (UID: \"40ef286d-ed76-4d2f-9457-6257d364a2ed\") "
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-model-cache" (OuterVolumeSpecName: "model-cache") pod "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" (UID: "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40ef286d-ed76-4d2f-9457-6257d364a2ed" (UID: "40ef286d-ed76-4d2f-9457-6257d364a2ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sx9l\" (UniqueName: \"kubernetes.io/projected/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-kube-api-access-8sx9l\") pod \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\" (UID: \"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00\") "
Sep 10 03:15:42.097351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "40ef286d-ed76-4d2f-9457-6257d364a2ed" (UID: "40ef286d-ed76-4d2f-9457-6257d364a2ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.097770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-home" (OuterVolumeSpecName: "home") pod "40ef286d-ed76-4d2f-9457-6257d364a2ed" (UID: "40ef286d-ed76-4d2f-9457-6257d364a2ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.097770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097686    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-home" (OuterVolumeSpecName: "home") pod "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" (UID: "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.097878 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097768    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" (UID: "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.097878 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097815    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee2514ee-183d-49f4-9d34-d7bcbab49933-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.097878 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097835    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.097878 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097848    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.097878 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097861    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.097878 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097874    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.098234 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097888    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cslnt\" (UniqueName: \"kubernetes.io/projected/ee2514ee-183d-49f4-9d34-d7bcbab49933-kube-api-access-cslnt\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.098234 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097903    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.098234 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097916    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.098234 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097952    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee2514ee-183d-49f4-9d34-d7bcbab49933-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.098234 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097967    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.098234 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.097982    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.099319 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.099294    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-dshm" (OuterVolumeSpecName: "dshm") pod "40ef286d-ed76-4d2f-9457-6257d364a2ed" (UID: "40ef286d-ed76-4d2f-9457-6257d364a2ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.099461 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.099443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-dshm" (OuterVolumeSpecName: "dshm") pod "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" (UID: "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:42.099703 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.099686    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ef286d-ed76-4d2f-9457-6257d364a2ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40ef286d-ed76-4d2f-9457-6257d364a2ed" (UID: "40ef286d-ed76-4d2f-9457-6257d364a2ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:15:42.099752 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.099731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" (UID: "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:15:42.099789 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.099746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-kube-api-access-8sx9l" (OuterVolumeSpecName: "kube-api-access-8sx9l") pod "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" (UID: "69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00"). InnerVolumeSpecName "kube-api-access-8sx9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:42.099948 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.099934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ef286d-ed76-4d2f-9457-6257d364a2ed-kube-api-access-wqz5x" (OuterVolumeSpecName: "kube-api-access-wqz5x") pod "40ef286d-ed76-4d2f-9457-6257d364a2ed" (UID: "40ef286d-ed76-4d2f-9457-6257d364a2ed"). InnerVolumeSpecName "kube-api-access-wqz5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:42.199291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.199148    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40ef286d-ed76-4d2f-9457-6257d364a2ed-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.199291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.199230    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqz5x\" (UniqueName: \"kubernetes.io/projected/40ef286d-ed76-4d2f-9457-6257d364a2ed-kube-api-access-wqz5x\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.199291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.199243    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sx9l\" (UniqueName: \"kubernetes.io/projected/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-kube-api-access-8sx9l\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.199291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.199252    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40ef286d-ed76-4d2f-9457-6257d364a2ed-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.199291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.199260    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.199291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.199267    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.199291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.199275    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.803503 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.803465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"
Sep 10 03:15:42.803503 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.803477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc" event={"ID":"40ef286d-ed76-4d2f-9457-6257d364a2ed","Type":"ContainerDied","Data":"29f9e57ee5c35ce09387117391cb05d241dcc746c58cd01a1bc8c905fff922f5"}
Sep 10 03:15:42.804044 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.803543    2580 scope.go:117] "RemoveContainer" containerID="f89905ac625505062f130f75beb7544072bf2de34577f163af0edb169c141d72"
Sep 10 03:15:42.805028 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.804973    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"
Sep 10 03:15:42.805112 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.805030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449" event={"ID":"69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00","Type":"ContainerDied","Data":"451bd13791427630e098474a2c15cd3814efbc07dd9ab13c6423956de0edf62e"}
Sep 10 03:15:42.806778 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.806753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q" event={"ID":"ee2514ee-183d-49f4-9d34-d7bcbab49933","Type":"ContainerDied","Data":"be0e2e8d8a125f217dc58027985b8ae0d003b8da776552c79bfed2ca28ff9378"}
Sep 10 03:15:42.806919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.806768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"
Sep 10 03:15:42.814258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.814233    2580 scope.go:117] "RemoveContainer" containerID="e79371ca404e7283aff3d40a4a96fb8624e4e09bd24e9e5f298ebb95ca618f25"
Sep 10 03:15:42.823225 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.823151    2580 scope.go:117] "RemoveContainer" containerID="ed552828165af892f79ad119c1d989cf23bb6df6f9b770ab58582da96377bf09"
Sep 10 03:15:42.830989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.830945    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"]
Sep 10 03:15:42.835910 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.835881    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/premium-w4-e6e6b8-kserve-54dcf74594-f8449"]
Sep 10 03:15:42.846575 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.846542    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"]
Sep 10 03:15:42.850466 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.850432    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q"]
Sep 10 03:15:42.862618 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.862528    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"]
Sep 10 03:15:42.866598 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:42.866568    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/unconfigured-w4-e6e6b8-kserve-b4575fd9d-q8mcc"]
Sep 10 03:15:43.326817 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:43.326734    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ef286d-ed76-4d2f-9457-6257d364a2ed" path="/var/lib/kubelet/pods/40ef286d-ed76-4d2f-9457-6257d364a2ed/volumes"
Sep 10 03:15:43.327185 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:43.327151    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" path="/var/lib/kubelet/pods/69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00/volumes"
Sep 10 03:15:43.327559 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:43.327545    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee2514ee-183d-49f4-9d34-d7bcbab49933" path="/var/lib/kubelet/pods/ee2514ee-183d-49f4-9d34-d7bcbab49933/volumes"
Sep 10 03:15:43.889513 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:43.889478    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"]
Sep 10 03:15:43.889987 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:43.889718    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy" containerID="cri-o://bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d" gracePeriod=30
Sep 10 03:15:46.887244 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:46.887188    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-w4-e6e6b8-kserve-5bbf9d8bc7-8ps5q" podUID="ee2514ee-183d-49f4-9d34-d7bcbab49933" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Sep 10 03:15:49.143834 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.143807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 03:15:49.174797 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.174718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61779f37-235f-4b0d-b030-8c4c125dac42-istiod-ca-cert\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.174950 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.174795    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-data\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.174950 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.174821    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61779f37-235f-4b0d-b030-8c4c125dac42-istio-podinfo\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.174950 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.174871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-istio-token\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.174950 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.174902    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-certs\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.174950 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.174949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-envoy\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.175207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-data" (OuterVolumeSpecName: "istio-data") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:49.175207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-socket\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.175317 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61779f37-235f-4b0d-b030-8c4c125dac42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:15:49.175317 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-credential-socket\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.175317 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b94xs\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-kube-api-access-b94xs\") pod \"61779f37-235f-4b0d-b030-8c4c125dac42\" (UID: \"61779f37-235f-4b0d-b030-8c4c125dac42\") "
Sep 10 03:15:49.175475 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175381    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:49.175525 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:49.175728 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175707    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.175799 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:49.175799 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175733    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.175799 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175758    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.175799 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.175772    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61779f37-235f-4b0d-b030-8c4c125dac42-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.177483 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.177458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-kube-api-access-b94xs" (OuterVolumeSpecName: "kube-api-access-b94xs") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "kube-api-access-b94xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:49.177668 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.177648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:49.177733 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.177716    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61779f37-235f-4b0d-b030-8c4c125dac42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:15:49.177798 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.177747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-istio-token" (OuterVolumeSpecName: "istio-token") pod "61779f37-235f-4b0d-b030-8c4c125dac42" (UID: "61779f37-235f-4b0d-b030-8c4c125dac42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:49.277082 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.277043    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61779f37-235f-4b0d-b030-8c4c125dac42-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.277082 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.277077    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.277082 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.277086    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.277082 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.277094    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61779f37-235f-4b0d-b030-8c4c125dac42-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.277403 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.277104    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b94xs\" (UniqueName: \"kubernetes.io/projected/61779f37-235f-4b0d-b030-8c4c125dac42-kube-api-access-b94xs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:15:49.847281 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.847238    2580 generic.go:356] "Generic (PLEG): container finished" podID="61779f37-235f-4b0d-b030-8c4c125dac42" containerID="bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d" exitCode=0
Sep 10 03:15:49.847470 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.847311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"
Sep 10 03:15:49.847470 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.847320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" event={"ID":"61779f37-235f-4b0d-b030-8c4c125dac42","Type":"ContainerDied","Data":"bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d"}
Sep 10 03:15:49.847470 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.847414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2" event={"ID":"61779f37-235f-4b0d-b030-8c4c125dac42","Type":"ContainerDied","Data":"ce74fd34ddcbbcdc74ecb2aa58fd3e983cb95edc615b0f74e5d63b09ea7786e0"}
Sep 10 03:15:49.847470 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.847439    2580 scope.go:117] "RemoveContainer" containerID="bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d"
Sep 10 03:15:49.856994 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.856970    2580 scope.go:117] "RemoveContainer" containerID="d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37"
Sep 10 03:15:49.866288 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.866265    2580 scope.go:117] "RemoveContainer" containerID="bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d"
Sep 10 03:15:49.866646 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:15:49.866620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d\": container with ID starting with bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d not found: ID does not exist" containerID="bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d"
Sep 10 03:15:49.866759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.866656    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d"} err="failed to get container status \"bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d\": rpc error: code = NotFound desc = could not find container \"bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d\": container with ID starting with bde6dfec1d17186cc87a7ed08e18251d96b190793e5a6b448c4df1bbd89b8e1d not found: ID does not exist"
Sep 10 03:15:49.866759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.866678    2580 scope.go:117] "RemoveContainer" containerID="d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37"
Sep 10 03:15:49.866759 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.866724    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"]
Sep 10 03:15:49.867017 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:15:49.866998    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37\": container with ID starting with d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37 not found: ID does not exist" containerID="d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37"
Sep 10 03:15:49.867085 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.867028    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37"} err="failed to get container status \"d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37\": rpc error: code = NotFound desc = could not find container \"d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37\": container with ID starting with d6b9d01d1cfea4e90df3f4dfa82fd9472881688c21a8fd9595f4f257639f7c37 not found: ID does not exist"
Sep 10 03:15:49.870229 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:49.870201    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e6e6b8-openshift-default-68d99bd7b4-b5jt2"]
Sep 10 03:15:51.325114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:51.325078    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" path="/var/lib/kubelet/pods/61779f37-235f-4b0d-b030-8c4c125dac42/volumes"
Sep 10 03:15:55.009059 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:55.009022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:15:57.814998 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.814960    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8015-6ce05d"]
Sep 10 03:15:57.815419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815411    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3e5a46-c6db-4ff0-ada8-e39a25644a6b" containerName="cleanup"
Sep 10 03:15:57.815471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815424    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3e5a46-c6db-4ff0-ada8-e39a25644a6b" containerName="cleanup"
Sep 10 03:15:57.815471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815435    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy"
Sep 10 03:15:57.815471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815440    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy"
Sep 10 03:15:57.815471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ef286d-ed76-4d2f-9457-6257d364a2ed" containerName="main"
Sep 10 03:15:57.815471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815458    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ef286d-ed76-4d2f-9457-6257d364a2ed" containerName="main"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815473    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446f3df1-0d84-42a1-b5e0-3158ec89c6f8" containerName="cleanup"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815478    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="446f3df1-0d84-42a1-b5e0-3158ec89c6f8" containerName="cleanup"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815487    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" containerName="main"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815492    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" containerName="main"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815504    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2514ee-183d-49f4-9d34-d7bcbab49933" containerName="main"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815509    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2514ee-183d-49f4-9d34-d7bcbab49933" containerName="main"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815567    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815576    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ef286d-ed76-4d2f-9457-6257d364a2ed" containerName="main"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815583    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="446f3df1-0d84-42a1-b5e0-3158ec89c6f8" containerName="cleanup"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815588    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3e5a46-c6db-4ff0-ada8-e39a25644a6b" containerName="cleanup"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815595    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fa6cbb-0c0c-4990-9d6d-ac5b5613ef00" containerName="main"
Sep 10 03:15:57.815635 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.815600    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2514ee-183d-49f4-9d34-d7bcbab49933" containerName="main"
Sep 10 03:15:57.820421 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.820400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:15:57.823033 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.823006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mtn2b\""
Sep 10 03:15:57.824100 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.824075    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8015-6ce05d"]
Sep 10 03:15:57.970057 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:57.970010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d7s2\" (UniqueName: \"kubernetes.io/projected/db6c81de-f906-449d-bf3e-6bbc6c180cf3-kube-api-access-9d7s2\") pod \"test-curl-8015-6ce05d\" (UID: \"db6c81de-f906-449d-bf3e-6bbc6c180cf3\") " pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:15:58.071504 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:58.071407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d7s2\" (UniqueName: \"kubernetes.io/projected/db6c81de-f906-449d-bf3e-6bbc6c180cf3-kube-api-access-9d7s2\") pod \"test-curl-8015-6ce05d\" (UID: \"db6c81de-f906-449d-bf3e-6bbc6c180cf3\") " pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:15:58.079029 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:58.078998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d7s2\" (UniqueName: \"kubernetes.io/projected/db6c81de-f906-449d-bf3e-6bbc6c180cf3-kube-api-access-9d7s2\") pod \"test-curl-8015-6ce05d\" (UID: \"db6c81de-f906-449d-bf3e-6bbc6c180cf3\") " pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:15:58.131937 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:58.131896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:15:58.404548 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:58.404520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8015-6ce05d"]
Sep 10 03:15:58.406394 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:15:58.406362    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb6c81de_f906_449d_bf3e_6bbc6c180cf3.slice/crio-f229ddfb40dad64f4bb1275e35a9759d8eeed6a0fa6bf273ebf38b8743184794 WatchSource:0}: Error finding container f229ddfb40dad64f4bb1275e35a9759d8eeed6a0fa6bf273ebf38b8743184794: Status 404 returned error can't find the container with id f229ddfb40dad64f4bb1275e35a9759d8eeed6a0fa6bf273ebf38b8743184794
Sep 10 03:15:58.895697 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:58.895663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8015-6ce05d" event={"ID":"db6c81de-f906-449d-bf3e-6bbc6c180cf3","Type":"ContainerStarted","Data":"e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071"}
Sep 10 03:15:58.895697 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:58.895706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8015-6ce05d" event={"ID":"db6c81de-f906-449d-bf3e-6bbc6c180cf3","Type":"ContainerStarted","Data":"f229ddfb40dad64f4bb1275e35a9759d8eeed6a0fa6bf273ebf38b8743184794"}
Sep 10 03:15:58.911134 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:58.911081    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8015-6ce05d" podStartSLOduration=1.9110641369999999 podStartE2EDuration="1.911064137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:15:58.909397962 +0000 UTC m=+1756.059297659" watchObservedRunningTime="2026-09-10 03:15:58.911064137 +0000 UTC m=+1756.060963835"
Sep 10 03:15:59.506547 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.506501    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8015-6ce05d"]
Sep 10 03:15:59.510203 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.510134    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8015-6ce05d"]
Sep 10 03:15:59.630269 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.630232    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8015-e36469"]
Sep 10 03:15:59.630653 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.630641    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy"
Sep 10 03:15:59.630701 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.630655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy"
Sep 10 03:15:59.630701 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.630677    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" containerName="test-curl-8015-6ce05d"
Sep 10 03:15:59.630701 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.630684    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" containerName="test-curl-8015-6ce05d"
Sep 10 03:15:59.630799 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.630739    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61779f37-235f-4b0d-b030-8c4c125dac42" containerName="istio-proxy"
Sep 10 03:15:59.630799 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.630753    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" containerName="test-curl-8015-6ce05d"
Sep 10 03:15:59.633842 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.633824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:15:59.639686 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.639662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8015-e36469"]
Sep 10 03:15:59.688100 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.688054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25m22\" (UniqueName: \"kubernetes.io/projected/bd25c24b-82a2-4017-a69c-e6317915f975-kube-api-access-25m22\") pod \"test-curl-8015-e36469\" (UID: \"bd25c24b-82a2-4017-a69c-e6317915f975\") " pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:15:59.789687 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.789645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25m22\" (UniqueName: \"kubernetes.io/projected/bd25c24b-82a2-4017-a69c-e6317915f975-kube-api-access-25m22\") pod \"test-curl-8015-e36469\" (UID: \"bd25c24b-82a2-4017-a69c-e6317915f975\") " pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:15:59.798006 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.797972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25m22\" (UniqueName: \"kubernetes.io/projected/bd25c24b-82a2-4017-a69c-e6317915f975-kube-api-access-25m22\") pod \"test-curl-8015-e36469\" (UID: \"bd25c24b-82a2-4017-a69c-e6317915f975\") " pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:15:59.901437 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:15:59.901349    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8015-6ce05d" container="test-curl-8015-6ce05d"
Sep 10 03:15:59.944426 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:15:59.944389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:16:00.214812 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:00.214722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8015-e36469"]
Sep 10 03:16:00.217741 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:16:00.217713    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd25c24b_82a2_4017_a69c_e6317915f975.slice/crio-7d2a2fb4091eb8810dea7108047a3db5eaae0ea8a998e0a81fd2cee50a840034 WatchSource:0}: Error finding container 7d2a2fb4091eb8810dea7108047a3db5eaae0ea8a998e0a81fd2cee50a840034: Status 404 returned error can't find the container with id 7d2a2fb4091eb8810dea7108047a3db5eaae0ea8a998e0a81fd2cee50a840034
Sep 10 03:16:00.908505 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:00.908443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8015-e36469" event={"ID":"bd25c24b-82a2-4017-a69c-e6317915f975","Type":"ContainerStarted","Data":"6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7"}
Sep 10 03:16:00.908505 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:00.908503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8015-e36469" event={"ID":"bd25c24b-82a2-4017-a69c-e6317915f975","Type":"ContainerStarted","Data":"7d2a2fb4091eb8810dea7108047a3db5eaae0ea8a998e0a81fd2cee50a840034"}
Sep 10 03:16:00.909076 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:00.908506    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8015-6ce05d" podUID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" containerName="test-curl-8015-6ce05d" containerID="cri-o://e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071" gracePeriod=2
Sep 10 03:16:00.910965 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:00.910937    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8015-6ce05d\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:16:00.925536 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:00.925475    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8015-e36469" podStartSLOduration=1.925459026 podStartE2EDuration="1.925459026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:16:00.925134743 +0000 UTC m=+1758.075034441" watchObservedRunningTime="2026-09-10 03:16:00.925459026 +0000 UTC m=+1758.075358715"
Sep 10 03:16:00.927369 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:00.927343    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8015-6ce05d\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:16:01.513371 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:01.513332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8015-e36469"]
Sep 10 03:16:01.516299 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:01.516268    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8015-e36469"]
Sep 10 03:16:01.913503 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:01.913472    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8015-e36469" container="test-curl-8015-e36469"
Sep 10 03:16:02.919390 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:02.919348    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8015-e36469" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" containerName="test-curl-8015-e36469" containerID="cri-o://6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7" gracePeriod=2
Sep 10 03:16:02.922281 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:02.922097    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8015-e36469\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:16:03.057013 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.056985    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:16:03.059323 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:03.059294    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8015-e36469\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:16:03.121823 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.121783    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d7s2\" (UniqueName: \"kubernetes.io/projected/db6c81de-f906-449d-bf3e-6bbc6c180cf3-kube-api-access-9d7s2\") pod \"db6c81de-f906-449d-bf3e-6bbc6c180cf3\" (UID: \"db6c81de-f906-449d-bf3e-6bbc6c180cf3\") "
Sep 10 03:16:03.123956 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.123929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6c81de-f906-449d-bf3e-6bbc6c180cf3-kube-api-access-9d7s2" (OuterVolumeSpecName: "kube-api-access-9d7s2") pod "db6c81de-f906-449d-bf3e-6bbc6c180cf3" (UID: "db6c81de-f906-449d-bf3e-6bbc6c180cf3"). InnerVolumeSpecName "kube-api-access-9d7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:16:03.223091 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.222972    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d7s2\" (UniqueName: \"kubernetes.io/projected/db6c81de-f906-449d-bf3e-6bbc6c180cf3-kube-api-access-9d7s2\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:16:03.324046 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:03.324009    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8015-e36469\" is forbidden: User \"system:node:ip-10-0-142-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-156.ec2.internal' and this object" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:16:03.327113 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.327079    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" path="/var/lib/kubelet/pods/db6c81de-f906-449d-bf3e-6bbc6c180cf3/volumes"
Sep 10 03:16:03.925940 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.925907    2580 generic.go:356] "Generic (PLEG): container finished" podID="db6c81de-f906-449d-bf3e-6bbc6c180cf3" containerID="e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071" exitCode=137
Sep 10 03:16:03.926389 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.925956    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-6ce05d"
Sep 10 03:16:03.926389 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.925984    2580 scope.go:117] "RemoveContainer" containerID="e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071"
Sep 10 03:16:03.934912 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.934889    2580 scope.go:117] "RemoveContainer" containerID="e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071"
Sep 10 03:16:03.935266 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:03.935245    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071\": container with ID starting with e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071 not found: ID does not exist" containerID="e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071"
Sep 10 03:16:03.935321 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:03.935278    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071"} err="failed to get container status \"e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071\": rpc error: code = NotFound desc = could not find container \"e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071\": container with ID starting with e079010b83143b1c5e015f37437bf7a65f981f2f534c1add3ddf8da1ea624071 not found: ID does not exist"
Sep 10 03:16:05.116272 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.116246    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:16:05.241092 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.240990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25m22\" (UniqueName: \"kubernetes.io/projected/bd25c24b-82a2-4017-a69c-e6317915f975-kube-api-access-25m22\") pod \"bd25c24b-82a2-4017-a69c-e6317915f975\" (UID: \"bd25c24b-82a2-4017-a69c-e6317915f975\") "
Sep 10 03:16:05.243232 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.243203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd25c24b-82a2-4017-a69c-e6317915f975-kube-api-access-25m22" (OuterVolumeSpecName: "kube-api-access-25m22") pod "bd25c24b-82a2-4017-a69c-e6317915f975" (UID: "bd25c24b-82a2-4017-a69c-e6317915f975"). InnerVolumeSpecName "kube-api-access-25m22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:16:05.325407 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.325336    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" path="/var/lib/kubelet/pods/bd25c24b-82a2-4017-a69c-e6317915f975/volumes"
Sep 10 03:16:05.342828 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.342796    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25m22\" (UniqueName: \"kubernetes.io/projected/bd25c24b-82a2-4017-a69c-e6317915f975-kube-api-access-25m22\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:16:05.938337 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.938303    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd25c24b-82a2-4017-a69c-e6317915f975" containerID="6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7" exitCode=137
Sep 10 03:16:05.938510 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.938347    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-e36469"
Sep 10 03:16:05.938510 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.938413    2580 scope.go:117] "RemoveContainer" containerID="6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7"
Sep 10 03:16:05.947108 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.947089    2580 scope.go:117] "RemoveContainer" containerID="6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7"
Sep 10 03:16:05.947438 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:16:05.947417    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7\": container with ID starting with 6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7 not found: ID does not exist" containerID="6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7"
Sep 10 03:16:05.947518 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:05.947445    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7"} err="failed to get container status \"6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7\": rpc error: code = NotFound desc = could not find container \"6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7\": container with ID starting with 6d231fb7ef035a7166dd92cbbeae03317d948f0c76cf964acf9416f08d0c73a7 not found: ID does not exist"
Sep 10 03:16:13.625366 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:13.621519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:16:17.221709 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.221675    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"]
Sep 10 03:16:17.222143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.222128    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" containerName="test-curl-8015-e36469"
Sep 10 03:16:17.222258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.222145    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" containerName="test-curl-8015-e36469"
Sep 10 03:16:17.222322 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.222270    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd25c24b-82a2-4017-a69c-e6317915f975" containerName="test-curl-8015-e36469"
Sep 10 03:16:17.225512 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.225487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.228822 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.228800    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"kube-root-ca.crt\""
Sep 10 03:16:17.230368 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.230347    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"openshift-service-ca.crt\""
Sep 10 03:16:17.230506 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.230369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"default-dockercfg-l2qbq\""
Sep 10 03:16:17.230506 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.230367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"rate-test-model-af7727-kserve-self-signed-certs\""
Sep 10 03:16:17.240285 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.240256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"]
Sep 10 03:16:17.260017 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.259981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zgw\" (UniqueName: \"kubernetes.io/projected/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-kube-api-access-h2zgw\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.260224 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.260083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tls-certs\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.260224 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.260122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-model-cache\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.260224 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.260211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-dshm\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.260406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.260305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tmp-dir\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.260406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.260363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-home\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.361787 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.361744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tmp-dir\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.361999 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.361826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-home\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.361999 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.361864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zgw\" (UniqueName: \"kubernetes.io/projected/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-kube-api-access-h2zgw\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.361999 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.361890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tls-certs\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.361999 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.361919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-model-cache\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.361999 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.361977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-dshm\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.362373 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.362349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tmp-dir\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.362441 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.362381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-model-cache\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.362441 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.362413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-home\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.364356 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.364336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-dshm\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.364443 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.364424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tls-certs\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.370371 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.370338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zgw\" (UniqueName: \"kubernetes.io/projected/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-kube-api-access-h2zgw\") pod \"rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") " pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.535984 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.535880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:17.810470 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:17.810445    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"]
Sep 10 03:16:17.812564 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:16:17.812529    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3ecc176_0f7e_4057_88ab_3abf896cf4c3.slice/crio-db6e0e6d8b03b1b0c83aeaeec6852b9eab027711af9e59f56710357828a663f2 WatchSource:0}: Error finding container db6e0e6d8b03b1b0c83aeaeec6852b9eab027711af9e59f56710357828a663f2: Status 404 returned error can't find the container with id db6e0e6d8b03b1b0c83aeaeec6852b9eab027711af9e59f56710357828a663f2
Sep 10 03:16:18.010051 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:18.009991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt" event={"ID":"c3ecc176-0f7e-4057-88ab-3abf896cf4c3","Type":"ContainerStarted","Data":"4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3"}
Sep 10 03:16:18.010051 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:18.010054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt" event={"ID":"c3ecc176-0f7e-4057-88ab-3abf896cf4c3","Type":"ContainerStarted","Data":"db6e0e6d8b03b1b0c83aeaeec6852b9eab027711af9e59f56710357828a663f2"}
Sep 10 03:16:18.028429 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:18.028375    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt" podStartSLOduration=1.028356048 podStartE2EDuration="1.028356048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:16:18.025756462 +0000 UTC m=+1775.175656162" watchObservedRunningTime="2026-09-10 03:16:18.028356048 +0000 UTC m=+1775.178255740"
Sep 10 03:16:26.006359 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:26.006319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:16:27.536826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:27.536768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:27.536826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:27.536839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:27.549799 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:27.549768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:28.074446 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:28.074412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:16:59.313250 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:16:59.313206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:17:13.607565 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:13.607524    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:17:23.606209 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:23.606152    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:17:29.793638 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:29.793601    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"]
Sep 10 03:17:34.319545 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:34.319508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:17:44.716240 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:44.716202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:17:44.814479 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:44.814426    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt" podUID="c3ecc176-0f7e-4057-88ab-3abf896cf4c3" containerName="main" containerID="cri-o://4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3" gracePeriod=45
Sep 10 03:17:45.059428 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.059401    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:17:45.142268 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2zgw\" (UniqueName: \"kubernetes.io/projected/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-kube-api-access-h2zgw\") pod \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") "
Sep 10 03:17:45.142459 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tls-certs\") pod \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") "
Sep 10 03:17:45.142459 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142310    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tmp-dir\") pod \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") "
Sep 10 03:17:45.142459 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-dshm\") pod \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") "
Sep 10 03:17:45.142596 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-home\") pod \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") "
Sep 10 03:17:45.142596 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-model-cache\") pod \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\" (UID: \"c3ecc176-0f7e-4057-88ab-3abf896cf4c3\") "
Sep 10 03:17:45.142704 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142650    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3ecc176-0f7e-4057-88ab-3abf896cf4c3" (UID: "c3ecc176-0f7e-4057-88ab-3abf896cf4c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:45.142798 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142769    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-home" (OuterVolumeSpecName: "home") pod "c3ecc176-0f7e-4057-88ab-3abf896cf4c3" (UID: "c3ecc176-0f7e-4057-88ab-3abf896cf4c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:45.142884 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3ecc176-0f7e-4057-88ab-3abf896cf4c3" (UID: "c3ecc176-0f7e-4057-88ab-3abf896cf4c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:45.142933 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142870    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:17:45.142933 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.142915    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:17:45.144625 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.144600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-dshm" (OuterVolumeSpecName: "dshm") pod "c3ecc176-0f7e-4057-88ab-3abf896cf4c3" (UID: "c3ecc176-0f7e-4057-88ab-3abf896cf4c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:45.144758 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.144633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3ecc176-0f7e-4057-88ab-3abf896cf4c3" (UID: "c3ecc176-0f7e-4057-88ab-3abf896cf4c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:17:45.144758 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.144656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-kube-api-access-h2zgw" (OuterVolumeSpecName: "kube-api-access-h2zgw") pod "c3ecc176-0f7e-4057-88ab-3abf896cf4c3" (UID: "c3ecc176-0f7e-4057-88ab-3abf896cf4c3"). InnerVolumeSpecName "kube-api-access-h2zgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:17:45.243856 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.243761    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:17:45.243856 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.243794    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2zgw\" (UniqueName: \"kubernetes.io/projected/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-kube-api-access-h2zgw\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:17:45.243856 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.243803    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:17:45.243856 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.243811    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ecc176-0f7e-4057-88ab-3abf896cf4c3-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:17:45.480981 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.480945    2580 generic.go:356] "Generic (PLEG): container finished" podID="c3ecc176-0f7e-4057-88ab-3abf896cf4c3" containerID="4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3" exitCode=0
Sep 10 03:17:45.481156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.480995    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"
Sep 10 03:17:45.481156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.481031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt" event={"ID":"c3ecc176-0f7e-4057-88ab-3abf896cf4c3","Type":"ContainerDied","Data":"4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3"}
Sep 10 03:17:45.481156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.481082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt" event={"ID":"c3ecc176-0f7e-4057-88ab-3abf896cf4c3","Type":"ContainerDied","Data":"db6e0e6d8b03b1b0c83aeaeec6852b9eab027711af9e59f56710357828a663f2"}
Sep 10 03:17:45.481156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.481099    2580 scope.go:117] "RemoveContainer" containerID="4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3"
Sep 10 03:17:45.490314 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.490295    2580 scope.go:117] "RemoveContainer" containerID="4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3"
Sep 10 03:17:45.490635 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:17:45.490608    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3\": container with ID starting with 4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3 not found: ID does not exist" containerID="4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3"
Sep 10 03:17:45.490705 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.490638    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3"} err="failed to get container status \"4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3\": rpc error: code = NotFound desc = could not find container \"4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3\": container with ID starting with 4fe672ffc753c17cf2e67cb1ea16cfbc368e3b3694e8daf52d0c50466789eec3 not found: ID does not exist"
Sep 10 03:17:45.499594 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.499531    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"]
Sep 10 03:17:45.501527 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:45.501503    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/rate-test-model-af7727-kserve-6ff6bbb4c-mhvvt"]
Sep 10 03:17:47.326061 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:17:47.326026    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ecc176-0f7e-4057-88ab-3abf896cf4c3" path="/var/lib/kubelet/pods/c3ecc176-0f7e-4057-88ab-3abf896cf4c3/volumes"
Sep 10 03:18:00.305241 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.305206    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"]
Sep 10 03:18:00.305904 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.305880    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ecc176-0f7e-4057-88ab-3abf896cf4c3" containerName="main"
Sep 10 03:18:00.306011 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.305904    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ecc176-0f7e-4057-88ab-3abf896cf4c3" containerName="main"
Sep 10 03:18:00.306071 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.306059    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ecc176-0f7e-4057-88ab-3abf896cf4c3" containerName="main"
Sep 10 03:18:00.309391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.309369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.313278 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.313249    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c970f3-openshift-default-dockercfg-mkkqz\""
Sep 10 03:18:00.321766 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.321740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"]
Sep 10 03:18:00.494463 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-podinfo\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494644 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-credential-socket\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494644 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-socket\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494644 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istiod-ca-cert\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494777 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-token\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494777 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-envoy\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494777 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-certs\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494777 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ws6v\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-kube-api-access-8ws6v\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.494938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.494812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-data\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.595612 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-socket\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.595805 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istiod-ca-cert\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.595805 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-token\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.595805 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-envoy\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.595805 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-certs\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.595805 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ws6v\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-kube-api-access-8ws6v\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.595805 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-data\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.596125 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-podinfo\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.596125 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.595880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-credential-socket\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.596257 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.596136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-socket\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.596257 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.596224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-data\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.596392 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.596359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-credential-socket\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.596512 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.596447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-certs\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.596512 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.596449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istiod-ca-cert\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.598265 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.598240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-envoy\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.598473 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.598453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-podinfo\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.607546 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.607524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ws6v\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-kube-api-access-8ws6v\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.607640 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.607621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-token\") pod \"e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") " pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.622717 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.622691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:00.907333 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.907303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"]
Sep 10 03:18:00.909063 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:18:00.909032    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bdfc88_4014_4f32_b78e_7f28df3aafa6.slice/crio-c1fc07fec755b60417aecc7179a4f707cddf654ebd928c8039abe51123ab3c10 WatchSource:0}: Error finding container c1fc07fec755b60417aecc7179a4f707cddf654ebd928c8039abe51123ab3c10: Status 404 returned error can't find the container with id c1fc07fec755b60417aecc7179a4f707cddf654ebd928c8039abe51123ab3c10
Sep 10 03:18:00.911337 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.911302    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:18:00.911411 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.911371    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:18:00.911452 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:00.911410    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:18:01.568426 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:01.568387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" event={"ID":"34bdfc88-4014-4f32-b78e-7f28df3aafa6","Type":"ContainerStarted","Data":"c98cba0b02f18989b173ceb201fdfc4efed030a9d66a26a3abbb9a4862df0d08"}
Sep 10 03:18:01.568830 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:01.568441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" event={"ID":"34bdfc88-4014-4f32-b78e-7f28df3aafa6","Type":"ContainerStarted","Data":"c1fc07fec755b60417aecc7179a4f707cddf654ebd928c8039abe51123ab3c10"}
Sep 10 03:18:01.593720 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:01.593665    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" podStartSLOduration=1.5936488130000002 podStartE2EDuration="1.593648813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:18:01.591554147 +0000 UTC m=+1878.741453846" watchObservedRunningTime="2026-09-10 03:18:01.593648813 +0000 UTC m=+1878.743548514"
Sep 10 03:18:01.623118 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:01.623076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:01.628904 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:01.628876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:02.574691 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:02.574657    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:02.575742 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:02.575719    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:05.610788 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.610741    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"]
Sep 10 03:18:05.614687 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.614658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:05.618062 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.618037    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c970f3-dockercfg-8lxk6\""
Sep 10 03:18:05.618216 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.618070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c970f3\""
Sep 10 03:18:05.626599 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.626570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"]
Sep 10 03:18:05.745701 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.745658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8ftw\" (UniqueName: \"kubernetes.io/projected/e34d3643-35e3-418a-bbd5-7d21a05afa08-kube-api-access-z8ftw\") pod \"payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:05.745701 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.745705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34d3643-35e3-418a-bbd5-7d21a05afa08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:05.846331 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.846292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34d3643-35e3-418a-bbd5-7d21a05afa08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:05.846498 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.846464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8ftw\" (UniqueName: \"kubernetes.io/projected/e34d3643-35e3-418a-bbd5-7d21a05afa08-kube-api-access-z8ftw\") pod \"payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:05.847143 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.847121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34d3643-35e3-418a-bbd5-7d21a05afa08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:05.854942 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.854905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8ftw\" (UniqueName: \"kubernetes.io/projected/e34d3643-35e3-418a-bbd5-7d21a05afa08-kube-api-access-z8ftw\") pod \"payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:05.928917 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:05.928798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:06.205082 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:06.205055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"]
Sep 10 03:18:06.206730 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:18:06.206703    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34d3643_35e3_418a_bbd5_7d21a05afa08.slice/crio-abbb4d6ca4757b2d8806332a03b7c9b65fa137d301b7232f4b3f6892c64b7908 WatchSource:0}: Error finding container abbb4d6ca4757b2d8806332a03b7c9b65fa137d301b7232f4b3f6892c64b7908: Status 404 returned error can't find the container with id abbb4d6ca4757b2d8806332a03b7c9b65fa137d301b7232f4b3f6892c64b7908
Sep 10 03:18:06.597989 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:06.597949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28" event={"ID":"e34d3643-35e3-418a-bbd5-7d21a05afa08","Type":"ContainerStarted","Data":"8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6"}
Sep 10 03:18:06.598202 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:06.598009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28" event={"ID":"e34d3643-35e3-418a-bbd5-7d21a05afa08","Type":"ContainerStarted","Data":"abbb4d6ca4757b2d8806332a03b7c9b65fa137d301b7232f4b3f6892c64b7908"}
Sep 10 03:18:06.598202 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:06.598039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:06.623035 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:06.622983    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28" podStartSLOduration=1.6229675860000001 podStartE2EDuration="1.622967586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:18:06.620280517 +0000 UTC m=+1883.770180229" watchObservedRunningTime="2026-09-10 03:18:06.622967586 +0000 UTC m=+1883.772867294"
Sep 10 03:18:11.753233 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:11.753195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:18:17.605545 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:17.605501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:18:23.694626 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:23.694592    2580 generic.go:356] "Generic (PLEG): container finished" podID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerID="c98cba0b02f18989b173ceb201fdfc4efed030a9d66a26a3abbb9a4862df0d08" exitCode=0
Sep 10 03:18:23.695081 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:23.694659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" event={"ID":"34bdfc88-4014-4f32-b78e-7f28df3aafa6","Type":"ContainerDied","Data":"c98cba0b02f18989b173ceb201fdfc4efed030a9d66a26a3abbb9a4862df0d08"}
Sep 10 03:18:23.695081 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:23.695073    2580 scope.go:117] "RemoveContainer" containerID="c98cba0b02f18989b173ceb201fdfc4efed030a9d66a26a3abbb9a4862df0d08"
Sep 10 03:18:23.696156 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:23.696124    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:18:23.696284 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:23.696221    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:18:23.696284 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:23.696255    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 03:18:24.702136 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:24.702098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" event={"ID":"34bdfc88-4014-4f32-b78e-7f28df3aafa6","Type":"ContainerStarted","Data":"a9b6fca60561c7384a3b0c462e7730fda41d96eac1d26037382e8a575ae0be90"}
Sep 10 03:18:25.624824 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:25.624780    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:25.625131 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:25.624961    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 10 03:18:25.625342 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:25.625312    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 10 03:18:26.623966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:26.623923    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 10 03:18:26.624611 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:26.624576    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 10 03:18:27.623434 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:27.623387    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 10 03:18:27.623677 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:27.623462    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 10 03:18:28.623965 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:28.623921    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 10 03:18:28.624655 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:28.624618    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 10 03:18:29.624395 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:29.623819    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 10 03:18:29.624395 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:29.623894    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 10 03:18:30.627881 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:30.627844    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:30.628412 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:30.628189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:30.629040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:30.629023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:18:33.663690 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:33.663650    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28_e34d3643-35e3-418a-bbd5-7d21a05afa08/payload-processing/0.log"
Sep 10 03:18:35.236945 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.236901    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"]
Sep 10 03:18:35.240091 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.240065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.244502 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.244471    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c9cef5\"/\"kube-root-ca.crt\""
Sep 10 03:18:35.244502 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.244496    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c9cef5\"/\"openshift-service-ca.crt\""
Sep 10 03:18:35.244689 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.244471    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c9cef5\"/\"ipp-route-c9cef5-kserve-self-signed-certs\""
Sep 10 03:18:35.244689 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.244513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c9cef5\"/\"default-dockercfg-kjgh6\""
Sep 10 03:18:35.254409 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.254379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"]
Sep 10 03:18:35.344633 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.344602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-home\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.344633 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.344633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-dshm\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.344919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.344745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tmp-dir\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.344919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.344805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78vm\" (UniqueName: \"kubernetes.io/projected/61d65dd9-d389-4386-9141-46f4b9f6e5eb-kube-api-access-n78vm\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.344919 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.344880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-model-cache\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.345047 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.344947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tls-certs\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446255 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tmp-dir\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446473 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78vm\" (UniqueName: \"kubernetes.io/projected/61d65dd9-d389-4386-9141-46f4b9f6e5eb-kube-api-access-n78vm\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446473 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-model-cache\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446600 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tls-certs\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446600 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-home\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446732 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-dshm\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446732 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-model-cache\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446732 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tmp-dir\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.446847 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.446785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-home\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.448891 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.448862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-dshm\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.449139 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.449118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tls-certs\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.457567 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.457533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78vm\" (UniqueName: \"kubernetes.io/projected/61d65dd9-d389-4386-9141-46f4b9f6e5eb-kube-api-access-n78vm\") pod \"ipp-route-c9cef5-kserve-7b79c5984c-m5w4q\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") " pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.551139 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.551048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:35.836435 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:35.836410    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"]
Sep 10 03:18:35.839123 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:18:35.839096    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d65dd9_d389_4386_9141_46f4b9f6e5eb.slice/crio-ca1ee3e1a83258ed5f4af8c32a03b0911d91582fcaa3f034a633e1b7b40d487b WatchSource:0}: Error finding container ca1ee3e1a83258ed5f4af8c32a03b0911d91582fcaa3f034a633e1b7b40d487b: Status 404 returned error can't find the container with id ca1ee3e1a83258ed5f4af8c32a03b0911d91582fcaa3f034a633e1b7b40d487b
Sep 10 03:18:36.777720 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:36.777682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q" event={"ID":"61d65dd9-d389-4386-9141-46f4b9f6e5eb","Type":"ContainerStarted","Data":"a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0"}
Sep 10 03:18:36.778158 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:36.777730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q" event={"ID":"61d65dd9-d389-4386-9141-46f4b9f6e5eb","Type":"ContainerStarted","Data":"ca1ee3e1a83258ed5f4af8c32a03b0911d91582fcaa3f034a633e1b7b40d487b"}
Sep 10 03:18:36.798448 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:36.798396    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q" podStartSLOduration=1.7983802070000001 podStartE2EDuration="1.798380207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:18:36.796247942 +0000 UTC m=+1913.946147651" watchObservedRunningTime="2026-09-10 03:18:36.798380207 +0000 UTC m=+1913.948279902"
Sep 10 03:18:39.948624 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:39.948583    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:18:43.793818 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:43.793783    2580 scope.go:117] "RemoveContainer" containerID="b3f646a0b835ebb0f0e49b82854f79c68a718c6a5e3d53c41a1a9336a6678042"
Sep 10 03:18:45.551445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:45.551389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:45.551445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:45.551463    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:45.564483 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:45.564453    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:18:45.840842 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:18:45.840749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:19:03.821600 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:03.821564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:19:09.764129 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:09.764098    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28_e34d3643-35e3-418a-bbd5-7d21a05afa08/payload-processing/0.log"
Sep 10 03:19:15.674583 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.674543    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"]
Sep 10 03:19:15.678631 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.678609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:15.681320 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.681293    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-be281e\""
Sep 10 03:19:15.681454 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.681406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-be281e-dockercfg-6mpwz\""
Sep 10 03:19:15.691127 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.691101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"]
Sep 10 03:19:15.863257 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.863221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1236918-8433-4299-8de5-ebde5bafd73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:15.863438 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.863316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwkv\" (UniqueName: \"kubernetes.io/projected/c1236918-8433-4299-8de5-ebde5bafd73d-kube-api-access-jtwkv\") pod \"payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:15.964573 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.964475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1236918-8433-4299-8de5-ebde5bafd73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:15.964573 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.964565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwkv\" (UniqueName: \"kubernetes.io/projected/c1236918-8433-4299-8de5-ebde5bafd73d-kube-api-access-jtwkv\") pod \"payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:15.965222 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.965198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1236918-8433-4299-8de5-ebde5bafd73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:15.972240 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.972212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwkv\" (UniqueName: \"kubernetes.io/projected/c1236918-8433-4299-8de5-ebde5bafd73d-kube-api-access-jtwkv\") pod \"payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:15.990239 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:15.990212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:16.262530 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:16.262504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"]
Sep 10 03:19:16.264896 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:19:16.264868    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1236918_8433_4299_8de5_ebde5bafd73d.slice/crio-7e30ce590d42d592fe0a039c7d13273a0d581c26ad46a3e91fd6168e070f1806 WatchSource:0}: Error finding container 7e30ce590d42d592fe0a039c7d13273a0d581c26ad46a3e91fd6168e070f1806: Status 404 returned error can't find the container with id 7e30ce590d42d592fe0a039c7d13273a0d581c26ad46a3e91fd6168e070f1806
Sep 10 03:19:17.005126 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:17.005088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62" event={"ID":"c1236918-8433-4299-8de5-ebde5bafd73d","Type":"ContainerStarted","Data":"74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0"}
Sep 10 03:19:17.005601 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:17.005263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:17.005601 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:17.005284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62" event={"ID":"c1236918-8433-4299-8de5-ebde5bafd73d","Type":"ContainerStarted","Data":"7e30ce590d42d592fe0a039c7d13273a0d581c26ad46a3e91fd6168e070f1806"}
Sep 10 03:19:17.024631 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:17.024563    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62" podStartSLOduration=2.024540946 podStartE2EDuration="2.024540946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:19:17.022959904 +0000 UTC m=+1954.172859603" watchObservedRunningTime="2026-09-10 03:19:17.024540946 +0000 UTC m=+1954.174440645"
Sep 10 03:19:25.111466 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:25.111430    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:19:28.017063 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:28.017028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:39.536025 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.535989    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"]
Sep 10 03:19:39.539884 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.539867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"
Sep 10 03:19:39.542797 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.542774    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:19:39.542934 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.542778    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:19:39.547581 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.547558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"]
Sep 10 03:19:39.614260 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.614224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50340b6a-a71d-45e0-8f3b-0f407ee28124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx\" (UID: \"50340b6a-a71d-45e0-8f3b-0f407ee28124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"
Sep 10 03:19:39.715808 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.715763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50340b6a-a71d-45e0-8f3b-0f407ee28124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx\" (UID: \"50340b6a-a71d-45e0-8f3b-0f407ee28124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"
Sep 10 03:19:39.716446 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.716427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50340b6a-a71d-45e0-8f3b-0f407ee28124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx\" (UID: \"50340b6a-a71d-45e0-8f3b-0f407ee28124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"
Sep 10 03:19:39.852669 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:39.852635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"
Sep 10 03:19:40.124949 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:40.124915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"]
Sep 10 03:19:40.143306 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:40.143275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx" event={"ID":"50340b6a-a71d-45e0-8f3b-0f407ee28124","Type":"ContainerStarted","Data":"d0773431f36d2b6dace40cf8446a8ffc178ccb2b167b4235478593794c198b7e"}
Sep 10 03:19:41.150293 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:41.150252    2580 generic.go:356] "Generic (PLEG): container finished" podID="50340b6a-a71d-45e0-8f3b-0f407ee28124" containerID="16e423d7e010231cd3f5229822dd332ac256c660083cea0e9b4614088ec6518e" exitCode=0
Sep 10 03:19:41.150293 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:41.150295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx" event={"ID":"50340b6a-a71d-45e0-8f3b-0f407ee28124","Type":"ContainerDied","Data":"16e423d7e010231cd3f5229822dd332ac256c660083cea0e9b4614088ec6518e"}
Sep 10 03:19:42.291265 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:42.291235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"
Sep 10 03:19:42.444647 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:42.444536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50340b6a-a71d-45e0-8f3b-0f407ee28124-maas-api-service-ca\") pod \"50340b6a-a71d-45e0-8f3b-0f407ee28124\" (UID: \"50340b6a-a71d-45e0-8f3b-0f407ee28124\") "
Sep 10 03:19:42.444972 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:42.444944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50340b6a-a71d-45e0-8f3b-0f407ee28124-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50340b6a-a71d-45e0-8f3b-0f407ee28124" (UID: "50340b6a-a71d-45e0-8f3b-0f407ee28124"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:19:42.546640 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:42.546581    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50340b6a-a71d-45e0-8f3b-0f407ee28124-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:19:43.161558 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:43.161528    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"
Sep 10 03:19:43.161742 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:43.161568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx" event={"ID":"50340b6a-a71d-45e0-8f3b-0f407ee28124","Type":"ContainerDied","Data":"d0773431f36d2b6dace40cf8446a8ffc178ccb2b167b4235478593794c198b7e"}
Sep 10 03:19:43.161742 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:43.161617    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0773431f36d2b6dace40cf8446a8ffc178ccb2b167b4235478593794c198b7e"
Sep 10 03:19:43.820438 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:43.820403    2580 scope.go:117] "RemoveContainer" containerID="e8b08542b33b134f6e208ea6112d167a5f6bb4891c841629a566491b1cbaf48b"
Sep 10 03:19:45.506756 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.506720    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"]
Sep 10 03:19:45.507251 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.506992    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62" podUID="c1236918-8433-4299-8de5-ebde5bafd73d" containerName="payload-pre-processing" containerID="cri-o://74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0" gracePeriod=30
Sep 10 03:19:45.648272 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.648234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"]
Sep 10 03:19:45.650611 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.650580    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-be281e-9j7vx"]
Sep 10 03:19:45.769298 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.769224    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:45.880274 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.880241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtwkv\" (UniqueName: \"kubernetes.io/projected/c1236918-8433-4299-8de5-ebde5bafd73d-kube-api-access-jtwkv\") pod \"c1236918-8433-4299-8de5-ebde5bafd73d\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") "
Sep 10 03:19:45.880454 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.880301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1236918-8433-4299-8de5-ebde5bafd73d-plugins-config-volume\") pod \"c1236918-8433-4299-8de5-ebde5bafd73d\" (UID: \"c1236918-8433-4299-8de5-ebde5bafd73d\") "
Sep 10 03:19:45.880734 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.880710    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1236918-8433-4299-8de5-ebde5bafd73d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1236918-8433-4299-8de5-ebde5bafd73d" (UID: "c1236918-8433-4299-8de5-ebde5bafd73d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:19:45.882421 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.882389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1236918-8433-4299-8de5-ebde5bafd73d-kube-api-access-jtwkv" (OuterVolumeSpecName: "kube-api-access-jtwkv") pod "c1236918-8433-4299-8de5-ebde5bafd73d" (UID: "c1236918-8433-4299-8de5-ebde5bafd73d"). InnerVolumeSpecName "kube-api-access-jtwkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:19:45.981524 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.981482    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwkv\" (UniqueName: \"kubernetes.io/projected/c1236918-8433-4299-8de5-ebde5bafd73d-kube-api-access-jtwkv\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:19:45.981524 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:45.981519    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1236918-8433-4299-8de5-ebde5bafd73d-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:19:46.177744 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.177711    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1236918-8433-4299-8de5-ebde5bafd73d" containerID="74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0" exitCode=0
Sep 10 03:19:46.177922 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.177760    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"
Sep 10 03:19:46.177922 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.177800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62" event={"ID":"c1236918-8433-4299-8de5-ebde5bafd73d","Type":"ContainerDied","Data":"74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0"}
Sep 10 03:19:46.177922 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.177855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62" event={"ID":"c1236918-8433-4299-8de5-ebde5bafd73d","Type":"ContainerDied","Data":"7e30ce590d42d592fe0a039c7d13273a0d581c26ad46a3e91fd6168e070f1806"}
Sep 10 03:19:46.177922 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.177884    2580 scope.go:117] "RemoveContainer" containerID="74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0"
Sep 10 03:19:46.188041 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.188019    2580 scope.go:117] "RemoveContainer" containerID="74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0"
Sep 10 03:19:46.188382 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:19:46.188363    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0\": container with ID starting with 74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0 not found: ID does not exist" containerID="74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0"
Sep 10 03:19:46.188444 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.188393    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0"} err="failed to get container status \"74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0\": rpc error: code = NotFound desc = could not find container \"74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0\": container with ID starting with 74bb6ec7fb9e9d41c6e8421c77b67fee362116a591d4957ebd82be73d79334f0 not found: ID does not exist"
Sep 10 03:19:46.203199 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.202892    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"]
Sep 10 03:19:46.209945 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:46.209925    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-be281e-c69d79cfd-t8h62"]
Sep 10 03:19:47.325608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:47.325572    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50340b6a-a71d-45e0-8f3b-0f407ee28124" path="/var/lib/kubelet/pods/50340b6a-a71d-45e0-8f3b-0f407ee28124/volumes"
Sep 10 03:19:47.326029 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:47.325955    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1236918-8433-4299-8de5-ebde5bafd73d" path="/var/lib/kubelet/pods/c1236918-8433-4299-8de5-ebde5bafd73d/volumes"
Sep 10 03:19:50.719198 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:50.719122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:19:56.021314 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.021268    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"]
Sep 10 03:19:56.021927 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.021906    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1236918-8433-4299-8de5-ebde5bafd73d" containerName="payload-pre-processing"
Sep 10 03:19:56.022040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.021930    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1236918-8433-4299-8de5-ebde5bafd73d" containerName="payload-pre-processing"
Sep 10 03:19:56.022040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.021947    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50340b6a-a71d-45e0-8f3b-0f407ee28124" containerName="revoke-keys"
Sep 10 03:19:56.022040 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.021955    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50340b6a-a71d-45e0-8f3b-0f407ee28124" containerName="revoke-keys"
Sep 10 03:19:56.022219 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.022100    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50340b6a-a71d-45e0-8f3b-0f407ee28124" containerName="revoke-keys"
Sep 10 03:19:56.022219 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.022113    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1236918-8433-4299-8de5-ebde5bafd73d" containerName="payload-pre-processing"
Sep 10 03:19:56.025419 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.025396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"
Sep 10 03:19:56.028132 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.028097    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:19:56.028295 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.028155    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:19:56.032748 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.032720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"]
Sep 10 03:19:56.086541 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.086501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c80c6-d725-477c-add0-0e7fa7691adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k\" (UID: \"c49c80c6-d725-477c-add0-0e7fa7691adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"
Sep 10 03:19:56.188015 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.187979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c80c6-d725-477c-add0-0e7fa7691adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k\" (UID: \"c49c80c6-d725-477c-add0-0e7fa7691adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"
Sep 10 03:19:56.188680 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.188658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c80c6-d725-477c-add0-0e7fa7691adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k\" (UID: \"c49c80c6-d725-477c-add0-0e7fa7691adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"
Sep 10 03:19:56.338435 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.338335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"
Sep 10 03:19:56.609424 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:56.609399    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"]
Sep 10 03:19:56.611648 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:19:56.611605    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49c80c6_d725_477c_add0_0e7fa7691adf.slice/crio-2b2042801a110a37a3812165f68c89a551d010e9dc4e4b4ee8742cd3acfca67d WatchSource:0}: Error finding container 2b2042801a110a37a3812165f68c89a551d010e9dc4e4b4ee8742cd3acfca67d: Status 404 returned error can't find the container with id 2b2042801a110a37a3812165f68c89a551d010e9dc4e4b4ee8742cd3acfca67d
Sep 10 03:19:57.240324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:57.240219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k" event={"ID":"c49c80c6-d725-477c-add0-0e7fa7691adf","Type":"ContainerStarted","Data":"ec63b204d7916ed69b27695524be7ab9ad3f173847f6de9763643759e2de1a1c"}
Sep 10 03:19:57.240324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:57.240273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k" event={"ID":"c49c80c6-d725-477c-add0-0e7fa7691adf","Type":"ContainerStarted","Data":"2b2042801a110a37a3812165f68c89a551d010e9dc4e4b4ee8742cd3acfca67d"}
Sep 10 03:19:57.256218 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:57.256141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k" podStartSLOduration=1.256123601 podStartE2EDuration="1.256123601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:19:57.254298332 +0000 UTC m=+1994.404198032" watchObservedRunningTime="2026-09-10 03:19:57.256123601 +0000 UTC m=+1994.406023298"
Sep 10 03:19:58.246689 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:58.246652    2580 generic.go:356] "Generic (PLEG): container finished" podID="c49c80c6-d725-477c-add0-0e7fa7691adf" containerID="ec63b204d7916ed69b27695524be7ab9ad3f173847f6de9763643759e2de1a1c" exitCode=0
Sep 10 03:19:58.247085 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:58.246737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k" event={"ID":"c49c80c6-d725-477c-add0-0e7fa7691adf","Type":"ContainerDied","Data":"ec63b204d7916ed69b27695524be7ab9ad3f173847f6de9763643759e2de1a1c"}
Sep 10 03:19:59.391626 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:59.391598    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"
Sep 10 03:19:59.525668 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:59.525578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c80c6-d725-477c-add0-0e7fa7691adf-maas-api-service-ca\") pod \"c49c80c6-d725-477c-add0-0e7fa7691adf\" (UID: \"c49c80c6-d725-477c-add0-0e7fa7691adf\") "
Sep 10 03:19:59.525944 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:59.525920    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49c80c6-d725-477c-add0-0e7fa7691adf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c49c80c6-d725-477c-add0-0e7fa7691adf" (UID: "c49c80c6-d725-477c-add0-0e7fa7691adf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:19:59.626744 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:19:59.626704    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c80c6-d725-477c-add0-0e7fa7691adf-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:20:00.259012 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:00.258976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k" event={"ID":"c49c80c6-d725-477c-add0-0e7fa7691adf","Type":"ContainerDied","Data":"2b2042801a110a37a3812165f68c89a551d010e9dc4e4b4ee8742cd3acfca67d"}
Sep 10 03:20:00.259012 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:00.259001    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"
Sep 10 03:20:00.259267 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:00.259029    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b2042801a110a37a3812165f68c89a551d010e9dc4e4b4ee8742cd3acfca67d"
Sep 10 03:20:09.216910 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:09.216869    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:20:11.648198 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:11.648124    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"]
Sep 10 03:20:11.651426 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:11.651399    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9cef5-n2s2k"]
Sep 10 03:20:13.325379 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:13.325344    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c49c80c6-d725-477c-add0-0e7fa7691adf" path="/var/lib/kubelet/pods/c49c80c6-d725-477c-add0-0e7fa7691adf/volumes"
Sep 10 03:20:17.000976 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:17.000942    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"]
Sep 10 03:20:21.525277 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:21.525238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:20:32.021493 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.021426    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q" podUID="61d65dd9-d389-4386-9141-46f4b9f6e5eb" containerName="main" containerID="cri-o://a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0" gracePeriod=45
Sep 10 03:20:32.283922 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.283849    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:20:32.364565 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-home\") pod \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") "
Sep 10 03:20:32.364763 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n78vm\" (UniqueName: \"kubernetes.io/projected/61d65dd9-d389-4386-9141-46f4b9f6e5eb-kube-api-access-n78vm\") pod \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") "
Sep 10 03:20:32.364763 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-dshm\") pod \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") "
Sep 10 03:20:32.364763 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tmp-dir\") pod \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") "
Sep 10 03:20:32.364935 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364774    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-model-cache\") pod \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") "
Sep 10 03:20:32.364935 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tls-certs\") pod \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\" (UID: \"61d65dd9-d389-4386-9141-46f4b9f6e5eb\") "
Sep 10 03:20:32.364935 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-home" (OuterVolumeSpecName: "home") pod "61d65dd9-d389-4386-9141-46f4b9f6e5eb" (UID: "61d65dd9-d389-4386-9141-46f4b9f6e5eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:20:32.365077 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.364994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61d65dd9-d389-4386-9141-46f4b9f6e5eb" (UID: "61d65dd9-d389-4386-9141-46f4b9f6e5eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:20:32.365077 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.365053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "61d65dd9-d389-4386-9141-46f4b9f6e5eb" (UID: "61d65dd9-d389-4386-9141-46f4b9f6e5eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:20:32.365215 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.365197    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:20:32.365278 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.365216    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:20:32.365278 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.365227    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:20:32.367047 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.367016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d65dd9-d389-4386-9141-46f4b9f6e5eb-kube-api-access-n78vm" (OuterVolumeSpecName: "kube-api-access-n78vm") pod "61d65dd9-d389-4386-9141-46f4b9f6e5eb" (UID: "61d65dd9-d389-4386-9141-46f4b9f6e5eb"). InnerVolumeSpecName "kube-api-access-n78vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:20:32.367150 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.367024    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-dshm" (OuterVolumeSpecName: "dshm") pod "61d65dd9-d389-4386-9141-46f4b9f6e5eb" (UID: "61d65dd9-d389-4386-9141-46f4b9f6e5eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:20:32.367150 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.367053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61d65dd9-d389-4386-9141-46f4b9f6e5eb" (UID: "61d65dd9-d389-4386-9141-46f4b9f6e5eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:20:32.434951 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.434912    2580 generic.go:356] "Generic (PLEG): container finished" podID="61d65dd9-d389-4386-9141-46f4b9f6e5eb" containerID="a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0" exitCode=0
Sep 10 03:20:32.434951 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.434955    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"
Sep 10 03:20:32.435233 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.434996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q" event={"ID":"61d65dd9-d389-4386-9141-46f4b9f6e5eb","Type":"ContainerDied","Data":"a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0"}
Sep 10 03:20:32.435233 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.435041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q" event={"ID":"61d65dd9-d389-4386-9141-46f4b9f6e5eb","Type":"ContainerDied","Data":"ca1ee3e1a83258ed5f4af8c32a03b0911d91582fcaa3f034a633e1b7b40d487b"}
Sep 10 03:20:32.435233 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.435059    2580 scope.go:117] "RemoveContainer" containerID="a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0"
Sep 10 03:20:32.444877 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.444857    2580 scope.go:117] "RemoveContainer" containerID="a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0"
Sep 10 03:20:32.445214 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:20:32.445197    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0\": container with ID starting with a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0 not found: ID does not exist" containerID="a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0"
Sep 10 03:20:32.445266 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.445222    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0"} err="failed to get container status \"a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0\": rpc error: code = NotFound desc = could not find container \"a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0\": container with ID starting with a6a81018cb1e1548ad3cdd3d9ae925e93bd8a4b11f13141deccad525d5dfa9d0 not found: ID does not exist"
Sep 10 03:20:32.457907 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.457870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"]
Sep 10 03:20:32.463872 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.463845    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c9cef5/ipp-route-c9cef5-kserve-7b79c5984c-m5w4q"]
Sep 10 03:20:32.466122 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.466103    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d65dd9-d389-4386-9141-46f4b9f6e5eb-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:20:32.466207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.466124    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78vm\" (UniqueName: \"kubernetes.io/projected/61d65dd9-d389-4386-9141-46f4b9f6e5eb-kube-api-access-n78vm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:20:32.466207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:32.466135    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d65dd9-d389-4386-9141-46f4b9f6e5eb-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:20:33.325768 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:20:33.325728    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61d65dd9-d389-4386-9141-46f4b9f6e5eb" path="/var/lib/kubelet/pods/61d65dd9-d389-4386-9141-46f4b9f6e5eb/volumes"
Sep 10 03:21:00.422000 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.421955    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"]
Sep 10 03:21:00.422654 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.422618    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d65dd9-d389-4386-9141-46f4b9f6e5eb" containerName="main"
Sep 10 03:21:00.422654 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.422641    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d65dd9-d389-4386-9141-46f4b9f6e5eb" containerName="main"
Sep 10 03:21:00.422782 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.422695    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49c80c6-d725-477c-add0-0e7fa7691adf" containerName="revoke-keys"
Sep 10 03:21:00.422782 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.422705    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49c80c6-d725-477c-add0-0e7fa7691adf" containerName="revoke-keys"
Sep 10 03:21:00.422881 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.422794    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c49c80c6-d725-477c-add0-0e7fa7691adf" containerName="revoke-keys"
Sep 10 03:21:00.422881 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.422816    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d65dd9-d389-4386-9141-46f4b9f6e5eb" containerName="main"
Sep 10 03:21:00.425971 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.425948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"
Sep 10 03:21:00.434017 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.433989    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:21:00.434017 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.433989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 03:21:00.449853 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.449820    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"]
Sep 10 03:21:00.456543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.456506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9dc40a-b028-4ccb-917f-76cec0e62310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78\" (UID: \"9f9dc40a-b028-4ccb-917f-76cec0e62310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"
Sep 10 03:21:00.557493 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.557446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9dc40a-b028-4ccb-917f-76cec0e62310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78\" (UID: \"9f9dc40a-b028-4ccb-917f-76cec0e62310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"
Sep 10 03:21:00.558124 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.558101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9dc40a-b028-4ccb-917f-76cec0e62310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78\" (UID: \"9f9dc40a-b028-4ccb-917f-76cec0e62310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"
Sep 10 03:21:00.736246 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:00.736135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"
Sep 10 03:21:01.009415 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:01.009384    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"]
Sep 10 03:21:01.011284 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:21:01.011255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9dc40a_b028_4ccb_917f_76cec0e62310.slice/crio-fa8c7c2e6497aac8e8afb94412a60470eec54948bc7cbda9e562452434c28ece WatchSource:0}: Error finding container fa8c7c2e6497aac8e8afb94412a60470eec54948bc7cbda9e562452434c28ece: Status 404 returned error can't find the container with id fa8c7c2e6497aac8e8afb94412a60470eec54948bc7cbda9e562452434c28ece
Sep 10 03:21:01.594344 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:01.594307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78" event={"ID":"9f9dc40a-b028-4ccb-917f-76cec0e62310","Type":"ContainerStarted","Data":"7356f485b52a45e1d925aef0f5461ad725252b71fbb4e423121dc13ca0a38764"}
Sep 10 03:21:01.594770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:01.594367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78" event={"ID":"9f9dc40a-b028-4ccb-917f-76cec0e62310","Type":"ContainerStarted","Data":"fa8c7c2e6497aac8e8afb94412a60470eec54948bc7cbda9e562452434c28ece"}
Sep 10 03:21:01.612013 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:01.611959    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78" podStartSLOduration=1.611942157 podStartE2EDuration="1.611942157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:21:01.611051461 +0000 UTC m=+2058.760951160" watchObservedRunningTime="2026-09-10 03:21:01.611942157 +0000 UTC m=+2058.761841855"
Sep 10 03:21:02.604061 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:02.604025    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f9dc40a-b028-4ccb-917f-76cec0e62310" containerID="7356f485b52a45e1d925aef0f5461ad725252b71fbb4e423121dc13ca0a38764" exitCode=0
Sep 10 03:21:02.604489 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:02.604120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78" event={"ID":"9f9dc40a-b028-4ccb-917f-76cec0e62310","Type":"ContainerDied","Data":"7356f485b52a45e1d925aef0f5461ad725252b71fbb4e423121dc13ca0a38764"}
Sep 10 03:21:03.747446 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:03.747411    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"
Sep 10 03:21:03.788537 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:03.788502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9dc40a-b028-4ccb-917f-76cec0e62310-maas-api-service-ca\") pod \"9f9dc40a-b028-4ccb-917f-76cec0e62310\" (UID: \"9f9dc40a-b028-4ccb-917f-76cec0e62310\") "
Sep 10 03:21:03.788981 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:03.788951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9dc40a-b028-4ccb-917f-76cec0e62310-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f9dc40a-b028-4ccb-917f-76cec0e62310" (UID: "9f9dc40a-b028-4ccb-917f-76cec0e62310"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:03.889690 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:03.889587    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9dc40a-b028-4ccb-917f-76cec0e62310-maas-api-service-ca\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:04.616057 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:04.616021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78" event={"ID":"9f9dc40a-b028-4ccb-917f-76cec0e62310","Type":"ContainerDied","Data":"fa8c7c2e6497aac8e8afb94412a60470eec54948bc7cbda9e562452434c28ece"}
Sep 10 03:21:04.616057 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:04.616040    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"
Sep 10 03:21:04.616302 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:04.616072    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa8c7c2e6497aac8e8afb94412a60470eec54948bc7cbda9e562452434c28ece"
Sep 10 03:21:10.908864 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:10.908770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"]
Sep 10 03:21:10.909364 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:10.908999    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28" podUID="e34d3643-35e3-418a-bbd5-7d21a05afa08" containerName="payload-processing" containerID="cri-o://8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6" gracePeriod=30
Sep 10 03:21:11.076692 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.076647    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"]
Sep 10 03:21:11.079648 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.079618    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c970f3-cwl78"]
Sep 10 03:21:11.164614 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.164528    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:21:11.262061 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.262012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8ftw\" (UniqueName: \"kubernetes.io/projected/e34d3643-35e3-418a-bbd5-7d21a05afa08-kube-api-access-z8ftw\") pod \"e34d3643-35e3-418a-bbd5-7d21a05afa08\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") "
Sep 10 03:21:11.262264 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.262078    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34d3643-35e3-418a-bbd5-7d21a05afa08-plugins-config-volume\") pod \"e34d3643-35e3-418a-bbd5-7d21a05afa08\" (UID: \"e34d3643-35e3-418a-bbd5-7d21a05afa08\") "
Sep 10 03:21:11.262556 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.262536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34d3643-35e3-418a-bbd5-7d21a05afa08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e34d3643-35e3-418a-bbd5-7d21a05afa08" (UID: "e34d3643-35e3-418a-bbd5-7d21a05afa08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:11.264230 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.264205    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34d3643-35e3-418a-bbd5-7d21a05afa08-kube-api-access-z8ftw" (OuterVolumeSpecName: "kube-api-access-z8ftw") pod "e34d3643-35e3-418a-bbd5-7d21a05afa08" (UID: "e34d3643-35e3-418a-bbd5-7d21a05afa08"). InnerVolumeSpecName "kube-api-access-z8ftw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:11.326095 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.326050    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f9dc40a-b028-4ccb-917f-76cec0e62310" path="/var/lib/kubelet/pods/9f9dc40a-b028-4ccb-917f-76cec0e62310/volumes"
Sep 10 03:21:11.363884 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.363846    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8ftw\" (UniqueName: \"kubernetes.io/projected/e34d3643-35e3-418a-bbd5-7d21a05afa08-kube-api-access-z8ftw\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.363884 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.363881    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34d3643-35e3-418a-bbd5-7d21a05afa08-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.654525 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.654489    2580 generic.go:356] "Generic (PLEG): container finished" podID="e34d3643-35e3-418a-bbd5-7d21a05afa08" containerID="8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6" exitCode=0
Sep 10 03:21:11.654750 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.654533    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"
Sep 10 03:21:11.654750 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.654579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28" event={"ID":"e34d3643-35e3-418a-bbd5-7d21a05afa08","Type":"ContainerDied","Data":"8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6"}
Sep 10 03:21:11.654750 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.654641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28" event={"ID":"e34d3643-35e3-418a-bbd5-7d21a05afa08","Type":"ContainerDied","Data":"abbb4d6ca4757b2d8806332a03b7c9b65fa137d301b7232f4b3f6892c64b7908"}
Sep 10 03:21:11.654750 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.654672    2580 scope.go:117] "RemoveContainer" containerID="8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6"
Sep 10 03:21:11.664276 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.664255    2580 scope.go:117] "RemoveContainer" containerID="8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6"
Sep 10 03:21:11.664578 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:21:11.664557    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6\": container with ID starting with 8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6 not found: ID does not exist" containerID="8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6"
Sep 10 03:21:11.664640 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.664581    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6"} err="failed to get container status \"8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6\": rpc error: code = NotFound desc = could not find container \"8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6\": container with ID starting with 8c6d5a996bc074bd7c760db135c7020ffd7b55c52f928fe7054673440f1005e6 not found: ID does not exist"
Sep 10 03:21:11.671744 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.671712    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"]
Sep 10 03:21:11.673939 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:11.673920    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c970f3-b65b4747-jfc28"]
Sep 10 03:21:13.332437 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:13.332385    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34d3643-35e3-418a-bbd5-7d21a05afa08" path="/var/lib/kubelet/pods/e34d3643-35e3-418a-bbd5-7d21a05afa08/volumes"
Sep 10 03:21:14.314335 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:14.314293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:21:17.579593 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:17.579551    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"]
Sep 10 03:21:17.580094 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:17.579800    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy" containerID="cri-o://a9b6fca60561c7384a3b0c462e7730fda41d96eac1d26037382e8a575ae0be90" gracePeriod=30
Sep 10 03:21:22.719956 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.719913    2580 generic.go:356] "Generic (PLEG): container finished" podID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerID="a9b6fca60561c7384a3b0c462e7730fda41d96eac1d26037382e8a575ae0be90" exitCode=0
Sep 10 03:21:22.720402 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.719979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" event={"ID":"34bdfc88-4014-4f32-b78e-7f28df3aafa6","Type":"ContainerDied","Data":"a9b6fca60561c7384a3b0c462e7730fda41d96eac1d26037382e8a575ae0be90"}
Sep 10 03:21:22.720402 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.720038    2580 scope.go:117] "RemoveContainer" containerID="c98cba0b02f18989b173ceb201fdfc4efed030a9d66a26a3abbb9a4862df0d08"
Sep 10 03:21:22.849126 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.849097    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:21:22.983674 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-data\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.983876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istiod-ca-cert\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.983876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983734    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-certs\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.983876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983761    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-token\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.983876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-socket\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.983876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-credential-socket\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.983876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983877    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-podinfo\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.984214 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ws6v\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-kube-api-access-8ws6v\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.984214 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-envoy\") pod \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\" (UID: \"34bdfc88-4014-4f32-b78e-7f28df3aafa6\") "
Sep 10 03:21:22.984214 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.983966    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-data" (OuterVolumeSpecName: "istio-data") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:22.984214 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:22.984432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:22.984432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:22.984432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984352    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:22.984432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984375    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:22.984432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984388    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:22.984432 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984400    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:22.984727 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.984447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:22.986186 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.986130    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-token" (OuterVolumeSpecName: "istio-token") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:22.986186 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.986160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:22.986338 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.986254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:21:22.986338 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:22.986286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-kube-api-access-8ws6v" (OuterVolumeSpecName: "kube-api-access-8ws6v") pod "34bdfc88-4014-4f32-b78e-7f28df3aafa6" (UID: "34bdfc88-4014-4f32-b78e-7f28df3aafa6"). InnerVolumeSpecName "kube-api-access-8ws6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:23.085192 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.085133    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:23.085192 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.085180    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:23.085192 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.085199    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ws6v\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-kube-api-access-8ws6v\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:23.085433 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.085212    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:23.085433 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.085224    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bdfc88-4014-4f32-b78e-7f28df3aafa6-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:23.726565 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.726535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"
Sep 10 03:21:23.727032 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.726567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m" event={"ID":"34bdfc88-4014-4f32-b78e-7f28df3aafa6","Type":"ContainerDied","Data":"c1fc07fec755b60417aecc7179a4f707cddf654ebd928c8039abe51123ab3c10"}
Sep 10 03:21:23.727032 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.726629    2580 scope.go:117] "RemoveContainer" containerID="a9b6fca60561c7384a3b0c462e7730fda41d96eac1d26037382e8a575ae0be90"
Sep 10 03:21:23.748629 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.748595    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"]
Sep 10 03:21:23.752702 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:23.752674    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c970f3-openshift-default-585c599b84-kqb2m"]
Sep 10 03:21:25.326005 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:25.325964    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" path="/var/lib/kubelet/pods/34bdfc88-4014-4f32-b78e-7f28df3aafa6/volumes"
Sep 10 03:21:28.612325 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.612268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"]
Sep 10 03:21:28.614073 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.614047    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-af7727-29816835-m9s6d"]
Sep 10 03:21:28.635497 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.635465    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"]
Sep 10 03:21:28.635777 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.635744    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr" podUID="7bc5140b-875d-42ba-a493-8524f424c2a5" containerName="payload-processing" containerID="cri-o://d6fe746bcf9ed4bef9f7c320aeeb1a33e4869161abc6c67cfca9906818613e91" gracePeriod=30
Sep 10 03:21:28.755564 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.755537    2580 generic.go:356] "Generic (PLEG): container finished" podID="7bc5140b-875d-42ba-a493-8524f424c2a5" containerID="d6fe746bcf9ed4bef9f7c320aeeb1a33e4869161abc6c67cfca9906818613e91" exitCode=0
Sep 10 03:21:28.755681 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.755620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr" event={"ID":"7bc5140b-875d-42ba-a493-8524f424c2a5","Type":"ContainerDied","Data":"d6fe746bcf9ed4bef9f7c320aeeb1a33e4869161abc6c67cfca9906818613e91"}
Sep 10 03:21:28.908840 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.908815    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:21:28.946402 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.946366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc5140b-875d-42ba-a493-8524f424c2a5-plugins-config-volume\") pod \"7bc5140b-875d-42ba-a493-8524f424c2a5\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") "
Sep 10 03:21:28.946600 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.946431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5grv6\" (UniqueName: \"kubernetes.io/projected/7bc5140b-875d-42ba-a493-8524f424c2a5-kube-api-access-5grv6\") pod \"7bc5140b-875d-42ba-a493-8524f424c2a5\" (UID: \"7bc5140b-875d-42ba-a493-8524f424c2a5\") "
Sep 10 03:21:28.946733 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.946710    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc5140b-875d-42ba-a493-8524f424c2a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bc5140b-875d-42ba-a493-8524f424c2a5" (UID: "7bc5140b-875d-42ba-a493-8524f424c2a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:28.948603 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:28.948578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc5140b-875d-42ba-a493-8524f424c2a5-kube-api-access-5grv6" (OuterVolumeSpecName: "kube-api-access-5grv6") pod "7bc5140b-875d-42ba-a493-8524f424c2a5" (UID: "7bc5140b-875d-42ba-a493-8524f424c2a5"). InnerVolumeSpecName "kube-api-access-5grv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:29.048026 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.047991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grv6\" (UniqueName: \"kubernetes.io/projected/7bc5140b-875d-42ba-a493-8524f424c2a5-kube-api-access-5grv6\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:29.048026 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.048021    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc5140b-875d-42ba-a493-8524f424c2a5-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:29.324834 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.324747    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3e5a46-c6db-4ff0-ada8-e39a25644a6b" path="/var/lib/kubelet/pods/de3e5a46-c6db-4ff0-ada8-e39a25644a6b/volumes"
Sep 10 03:21:29.761737 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.761706    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"
Sep 10 03:21:29.762278 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.761752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr" event={"ID":"7bc5140b-875d-42ba-a493-8524f424c2a5","Type":"ContainerDied","Data":"8631db82b4144e9904c8fe14d764c9ece50e59d26b271efa0d2c58a054c0f1bd"}
Sep 10 03:21:29.762278 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.761821    2580 scope.go:117] "RemoveContainer" containerID="d6fe746bcf9ed4bef9f7c320aeeb1a33e4869161abc6c67cfca9906818613e91"
Sep 10 03:21:29.782024 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.781985    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"]
Sep 10 03:21:29.786568 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:29.786540    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7727-6794fb697f-8mmdr"]
Sep 10 03:21:31.325216 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:31.325179    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc5140b-875d-42ba-a493-8524f424c2a5" path="/var/lib/kubelet/pods/7bc5140b-875d-42ba-a493-8524f424c2a5/volumes"
Sep 10 03:21:31.808471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:31.808439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:21:43.864041 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:43.864006    2580 scope.go:117] "RemoveContainer" containerID="9a05395d5c4b08ed19772a1408ccb00e3e83fce53db8eeef1bc6285f19b234c7"
Sep 10 03:21:46.214534 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.214497    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"]
Sep 10 03:21:46.215088 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.214715    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv" podUID="67b470f1-aa77-40ad-a508-fb9d048db979" containerName="payload-processing" containerID="cri-o://127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52" gracePeriod=30
Sep 10 03:21:46.219978 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.219945    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"]
Sep 10 03:21:46.220252 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.220206    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr" podUID="fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" containerName="payload-pre-processing" containerID="cri-o://fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6" gracePeriod=30
Sep 10 03:21:46.480544 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.480467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:21:46.510689 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.510664    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:21:46.618352 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.618297    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tkrw\" (UniqueName: \"kubernetes.io/projected/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-kube-api-access-2tkrw\") pod \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") "
Sep 10 03:21:46.618352 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.618371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b470f1-aa77-40ad-a508-fb9d048db979-plugins-config-volume\") pod \"67b470f1-aa77-40ad-a508-fb9d048db979\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") "
Sep 10 03:21:46.618639 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.618403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5qg2\" (UniqueName: \"kubernetes.io/projected/67b470f1-aa77-40ad-a508-fb9d048db979-kube-api-access-q5qg2\") pod \"67b470f1-aa77-40ad-a508-fb9d048db979\" (UID: \"67b470f1-aa77-40ad-a508-fb9d048db979\") "
Sep 10 03:21:46.618639 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.618491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-plugins-config-volume\") pod \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\" (UID: \"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6\") "
Sep 10 03:21:46.618785 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.618757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b470f1-aa77-40ad-a508-fb9d048db979-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67b470f1-aa77-40ad-a508-fb9d048db979" (UID: "67b470f1-aa77-40ad-a508-fb9d048db979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:46.618931 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.618901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" (UID: "fd6a7f0f-b2d3-45b1-a215-6001a2072ea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:46.620573 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.620549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-kube-api-access-2tkrw" (OuterVolumeSpecName: "kube-api-access-2tkrw") pod "fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" (UID: "fd6a7f0f-b2d3-45b1-a215-6001a2072ea6"). InnerVolumeSpecName "kube-api-access-2tkrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:46.620654 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.620622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b470f1-aa77-40ad-a508-fb9d048db979-kube-api-access-q5qg2" (OuterVolumeSpecName: "kube-api-access-q5qg2") pod "67b470f1-aa77-40ad-a508-fb9d048db979" (UID: "67b470f1-aa77-40ad-a508-fb9d048db979"). InnerVolumeSpecName "kube-api-access-q5qg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:46.719279 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.719224    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:46.719279 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.719271    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tkrw\" (UniqueName: \"kubernetes.io/projected/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6-kube-api-access-2tkrw\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:46.719279 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.719286    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b470f1-aa77-40ad-a508-fb9d048db979-plugins-config-volume\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:46.719543 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.719298    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qg2\" (UniqueName: \"kubernetes.io/projected/67b470f1-aa77-40ad-a508-fb9d048db979-kube-api-access-q5qg2\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:46.854401 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.854368    2580 generic.go:356] "Generic (PLEG): container finished" podID="67b470f1-aa77-40ad-a508-fb9d048db979" containerID="127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52" exitCode=0
Sep 10 03:21:46.854603 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.854409    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"
Sep 10 03:21:46.854603 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.854454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv" event={"ID":"67b470f1-aa77-40ad-a508-fb9d048db979","Type":"ContainerDied","Data":"127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52"}
Sep 10 03:21:46.854603 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.854517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv" event={"ID":"67b470f1-aa77-40ad-a508-fb9d048db979","Type":"ContainerDied","Data":"5eaf56e2d73537b8aa68f7f5dbb173271ac7bb8b7647f661056e74496db68ae6"}
Sep 10 03:21:46.854603 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.854554    2580 scope.go:117] "RemoveContainer" containerID="127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52"
Sep 10 03:21:46.855894 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.855875    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" containerID="fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6" exitCode=0
Sep 10 03:21:46.855995 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.855922    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"
Sep 10 03:21:46.855995 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.855962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr" event={"ID":"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6","Type":"ContainerDied","Data":"fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6"}
Sep 10 03:21:46.856077 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.856024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr" event={"ID":"fd6a7f0f-b2d3-45b1-a215-6001a2072ea6","Type":"ContainerDied","Data":"3772fb2d5d91c27b8151f3d58169f5159cd701ff6ce6dcbb025c17f89fec7b64"}
Sep 10 03:21:46.864722 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.864699    2580 scope.go:117] "RemoveContainer" containerID="127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52"
Sep 10 03:21:46.865050 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:21:46.865027    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52\": container with ID starting with 127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52 not found: ID does not exist" containerID="127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52"
Sep 10 03:21:46.865111 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.865070    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52"} err="failed to get container status \"127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52\": rpc error: code = NotFound desc = could not find container \"127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52\": container with ID starting with 127318544d49d6d80e92e7f3d955692ea892fbdba61b7ec0f7602ace7b416b52 not found: ID does not exist"
Sep 10 03:21:46.865111 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.865090    2580 scope.go:117] "RemoveContainer" containerID="fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6"
Sep 10 03:21:46.873987 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.873969    2580 scope.go:117] "RemoveContainer" containerID="fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6"
Sep 10 03:21:46.874331 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:21:46.874307    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6\": container with ID starting with fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6 not found: ID does not exist" containerID="fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6"
Sep 10 03:21:46.874433 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.874336    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6"} err="failed to get container status \"fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6\": rpc error: code = NotFound desc = could not find container \"fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6\": container with ID starting with fe8a1e3fc222b41eec12297f18a5f8b304e7d8656159caccc37be942b6de97c6 not found: ID does not exist"
Sep 10 03:21:46.880851 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.880815    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"]
Sep 10 03:21:46.882606 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.882585    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7daa14-5767b64bf7-fplrv"]
Sep 10 03:21:46.892915 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.892886    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"]
Sep 10 03:21:46.898640 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:46.898612    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7daa14-84f97f9c66-tcwmr"]
Sep 10 03:21:47.331372 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:47.331155    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b470f1-aa77-40ad-a508-fb9d048db979" path="/var/lib/kubelet/pods/67b470f1-aa77-40ad-a508-fb9d048db979/volumes"
Sep 10 03:21:47.331792 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:47.331776    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" path="/var/lib/kubelet/pods/fd6a7f0f-b2d3-45b1-a215-6001a2072ea6/volumes"
Sep 10 03:21:49.205835 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:49.205784    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:21:52.848921 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:52.848886    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"]
Sep 10 03:21:52.849474 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:52.849124    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy" containerID="cri-o://1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93" gracePeriod=30
Sep 10 03:21:58.107946 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.107910    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:21:58.228324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-podinfo\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-socket\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228324 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228319    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-data\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-envoy\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istiod-ca-cert\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-token\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-certs\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-credential-socket\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7jj7\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-kube-api-access-w7jj7\") pod \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\" (UID: \"6b3b764b-ceea-4f90-a810-d2dc9bfb9449\") "
Sep 10 03:21:58.228617 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:58.228966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:58.228966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:58.228966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:58.228966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:58.228966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228941    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-credential-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.228966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228958    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-socket\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.229342 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228972    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-data\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.229342 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228986    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istiod-ca-cert\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.229342 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.228998    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-workload-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.230648 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.230620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 03:21:58.230770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.230731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:58.230770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.230741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:21:58.230869 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.230824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-kube-api-access-w7jj7" (OuterVolumeSpecName: "kube-api-access-w7jj7") pod "6b3b764b-ceea-4f90-a810-d2dc9bfb9449" (UID: "6b3b764b-ceea-4f90-a810-d2dc9bfb9449"). InnerVolumeSpecName "kube-api-access-w7jj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:58.308541 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.308501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:21:58.329446 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.329417    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-podinfo\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.329446 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.329443    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-envoy\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.329585 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.329455    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-istio-token\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.329585 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.329466    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7jj7\" (UniqueName: \"kubernetes.io/projected/6b3b764b-ceea-4f90-a810-d2dc9bfb9449-kube-api-access-w7jj7\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:21:58.929028 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.928988    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerID="1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93" exitCode=0
Sep 10 03:21:58.929241 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.929038    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"
Sep 10 03:21:58.929241 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.929037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" event={"ID":"6b3b764b-ceea-4f90-a810-d2dc9bfb9449","Type":"ContainerDied","Data":"1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93"}
Sep 10 03:21:58.929241 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.929157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x" event={"ID":"6b3b764b-ceea-4f90-a810-d2dc9bfb9449","Type":"ContainerDied","Data":"c5c57e1b3d0cf4ec7a0631312be3ecc79f45be62e548ceccf974a5c624177d51"}
Sep 10 03:21:58.929241 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.929209    2580 scope.go:117] "RemoveContainer" containerID="1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93"
Sep 10 03:21:58.940312 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.940291    2580 scope.go:117] "RemoveContainer" containerID="be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a"
Sep 10 03:21:58.950737 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.950711    2580 scope.go:117] "RemoveContainer" containerID="1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93"
Sep 10 03:21:58.951117 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:21:58.951094    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93\": container with ID starting with 1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93 not found: ID does not exist" containerID="1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93"
Sep 10 03:21:58.951251 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.951126    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93"} err="failed to get container status \"1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93\": rpc error: code = NotFound desc = could not find container \"1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93\": container with ID starting with 1bab2f6e4783b4162d7594bc5d2a44e74a72d58cc55b57a2e340338b72a5cd93 not found: ID does not exist"
Sep 10 03:21:58.951251 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.951144    2580 scope.go:117] "RemoveContainer" containerID="be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a"
Sep 10 03:21:58.951494 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:21:58.951476    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a\": container with ID starting with be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a not found: ID does not exist" containerID="be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a"
Sep 10 03:21:58.951541 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.951502    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a"} err="failed to get container status \"be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a\": rpc error: code = NotFound desc = could not find container \"be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a\": container with ID starting with be02c60f0d1786644555337914a5c29359339a07e2835bb5ef2e5f585cbc244a not found: ID does not exist"
Sep 10 03:21:58.952077 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.952057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"]
Sep 10 03:21:58.955623 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:58.955599    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7daa14-openshift-default-787b96ccf4-bx96x"]
Sep 10 03:21:59.327713 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:21:59.327624    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" path="/var/lib/kubelet/pods/6b3b764b-ceea-4f90-a810-d2dc9bfb9449/volumes"
Sep 10 03:23:20.405273 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:23:20.405232    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:23:44.120258 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:23:44.120216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:23:52.320331 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:23:52.320293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:24:01.610685 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:24:01.610647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:24:16.514189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:24:16.514088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:24:25.027803 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:24:25.027760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:25:43.934886 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:25:43.934847    2580 scope.go:117] "RemoveContainer" containerID="16e423d7e010231cd3f5229822dd332ac256c660083cea0e9b4614088ec6518e"
Sep 10 03:25:53.120903 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:25:53.120869    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:26:00.209729 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:00.209693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:26:10.009316 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:10.009279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:26:15.915337 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915299    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"]
Sep 10 03:26:15.915938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b470f1-aa77-40ad-a508-fb9d048db979" containerName="payload-processing"
Sep 10 03:26:15.915938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915869    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b470f1-aa77-40ad-a508-fb9d048db979" containerName="payload-processing"
Sep 10 03:26:15.915938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915882    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy"
Sep 10 03:26:15.915938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915891    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy"
Sep 10 03:26:15.915938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915903    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34d3643-35e3-418a-bbd5-7d21a05afa08" containerName="payload-processing"
Sep 10 03:26:15.915938 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915912    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34d3643-35e3-418a-bbd5-7d21a05afa08" containerName="payload-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915958    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc5140b-875d-42ba-a493-8524f424c2a5" containerName="payload-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915967    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc5140b-875d-42ba-a493-8524f424c2a5" containerName="payload-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915988    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.915996    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" containerName="payload-pre-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916026    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" containerName="payload-pre-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916036    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f9dc40a-b028-4ccb-917f-76cec0e62310" containerName="revoke-keys"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916044    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f9dc40a-b028-4ccb-917f-76cec0e62310" containerName="revoke-keys"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916057    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916065    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916077    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916085    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916189    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f9dc40a-b028-4ccb-917f-76cec0e62310" containerName="revoke-keys"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916204    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916213    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc5140b-875d-42ba-a493-8524f424c2a5" containerName="payload-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916225    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd6a7f0f-b2d3-45b1-a215-6001a2072ea6" containerName="payload-pre-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916234    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3b764b-ceea-4f90-a810-d2dc9bfb9449" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916243    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b470f1-aa77-40ad-a508-fb9d048db979" containerName="payload-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916250    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916258    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34d3643-35e3-418a-bbd5-7d21a05afa08" containerName="payload-processing"
Sep 10 03:26:15.916307 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.916265    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34bdfc88-4014-4f32-b78e-7f28df3aafa6" containerName="istio-proxy"
Sep 10 03:26:15.919823 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.919800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:15.922817 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.922793    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-082ad071-kserve-self-signed-certs\""
Sep 10 03:26:15.925893 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.925872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"]
Sep 10 03:26:15.948372 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.948341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dzz\" (UniqueName: \"kubernetes.io/projected/a2520210-f8cc-47fe-a2b6-dcef3604c80a-kube-api-access-x8dzz\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:15.948508 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.948381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-home\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:15.948508 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.948397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-model-cache\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:15.948508 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.948430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tls-certs\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:15.948508 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.948452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-dshm\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:15.948508 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:15.948478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tmp-dir\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.048973 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.048937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-home\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.048973 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.048972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-model-cache\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.049248 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.048998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tls-certs\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.049248 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.049025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-dshm\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.049248 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.049060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tmp-dir\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.049248 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.049138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dzz\" (UniqueName: \"kubernetes.io/projected/a2520210-f8cc-47fe-a2b6-dcef3604c80a-kube-api-access-x8dzz\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.049472 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.049405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-home\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.049472 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.049458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-model-cache\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.049554 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.049488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tmp-dir\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.051243 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.051217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-dshm\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.051557 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.051541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tls-certs\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.057511 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.057489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dzz\" (UniqueName: \"kubernetes.io/projected/a2520210-f8cc-47fe-a2b6-dcef3604c80a-kube-api-access-x8dzz\") pod \"e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") " pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.230537 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.230448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:16.499095 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:16.499069    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"]
Sep 10 03:26:16.500990 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:26:16.500964    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2520210_f8cc_47fe_a2b6_dcef3604c80a.slice/crio-a942d413d62f03fe322c9eb665a3c0229333eb51afbd84590c6ab409527d2c1a WatchSource:0}: Error finding container a942d413d62f03fe322c9eb665a3c0229333eb51afbd84590c6ab409527d2c1a: Status 404 returned error can't find the container with id a942d413d62f03fe322c9eb665a3c0229333eb51afbd84590c6ab409527d2c1a
Sep 10 03:26:17.249380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:17.249348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l" event={"ID":"a2520210-f8cc-47fe-a2b6-dcef3604c80a","Type":"ContainerStarted","Data":"50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2"}
Sep 10 03:26:17.249785 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:17.249393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l" event={"ID":"a2520210-f8cc-47fe-a2b6-dcef3604c80a","Type":"ContainerStarted","Data":"a942d413d62f03fe322c9eb665a3c0229333eb51afbd84590c6ab409527d2c1a"}
Sep 10 03:26:17.268017 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:17.267973    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l" podStartSLOduration=2.2679573140000002 podStartE2EDuration="2.267957314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:26:17.266885013 +0000 UTC m=+2374.416784722" watchObservedRunningTime="2026-09-10 03:26:17.267957314 +0000 UTC m=+2374.417857012"
Sep 10 03:26:18.614862 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:18.614824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:26:26.230699 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:26.230654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:26.230699 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:26.230710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:26.243747 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:26.243712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:26.304288 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:26.304255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:26:43.954223 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:43.954191    2580 scope.go:117] "RemoveContainer" containerID="ec63b204d7916ed69b27695524be7ab9ad3f173847f6de9763643759e2de1a1c"
Sep 10 03:26:44.214686 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.214607    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"]
Sep 10 03:26:44.219861 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.219839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.222647 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.222625    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-082ad071-kserve-self-signed-certs\""
Sep 10 03:26:44.228826 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.228802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"]
Sep 10 03:26:44.315345 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.315305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tmp-dir\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.315345 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.315351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-model-cache\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.315621 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.315383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-home\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.315621 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.315502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dlc\" (UniqueName: \"kubernetes.io/projected/72eef7bf-cd1b-4bf8-b2ee-79142510466b-kube-api-access-45dlc\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.315621 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.315556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tls-certs\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.315621 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.315585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-dshm\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416151 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dlc\" (UniqueName: \"kubernetes.io/projected/72eef7bf-cd1b-4bf8-b2ee-79142510466b-kube-api-access-45dlc\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416151 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tls-certs\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416436 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-dshm\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416436 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tmp-dir\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416436 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-model-cache\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416615 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-home\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416780 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-model-cache\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416913 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tmp-dir\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.416913 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.416806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-home\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.418474 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.418455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-dshm\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.418770 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.418751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tls-certs\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.423966 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.423942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dlc\" (UniqueName: \"kubernetes.io/projected/72eef7bf-cd1b-4bf8-b2ee-79142510466b-kube-api-access-45dlc\") pod \"e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") " pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.531116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.531025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:44.802736 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:44.802672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"]
Sep 10 03:26:44.805340 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:26:44.805310    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72eef7bf_cd1b_4bf8_b2ee_79142510466b.slice/crio-0c7b728b89b813b7a8313aa98a8eb095791049c17ff0c7355b827a984297e352 WatchSource:0}: Error finding container 0c7b728b89b813b7a8313aa98a8eb095791049c17ff0c7355b827a984297e352: Status 404 returned error can't find the container with id 0c7b728b89b813b7a8313aa98a8eb095791049c17ff0c7355b827a984297e352
Sep 10 03:26:45.389996 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:45.389955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg" event={"ID":"72eef7bf-cd1b-4bf8-b2ee-79142510466b","Type":"ContainerStarted","Data":"bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25"}
Sep 10 03:26:45.389996 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:45.390003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg" event={"ID":"72eef7bf-cd1b-4bf8-b2ee-79142510466b","Type":"ContainerStarted","Data":"0c7b728b89b813b7a8313aa98a8eb095791049c17ff0c7355b827a984297e352"}
Sep 10 03:26:45.408099 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:45.408048    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg" podStartSLOduration=1.408029689 podStartE2EDuration="1.408029689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:26:45.406943274 +0000 UTC m=+2402.556842978" watchObservedRunningTime="2026-09-10 03:26:45.408029689 +0000 UTC m=+2402.557929387"
Sep 10 03:26:47.215016 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:47.214978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:26:54.531639 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:54.531602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:54.531639 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:54.531649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:54.544534 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:54.544509    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:26:55.448815 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:26:55.448786    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:27:09.555010 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:09.554931    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"]
Sep 10 03:27:11.247281 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:11.247248    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"]
Sep 10 03:27:12.018087 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:12.018049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:27:19.206795 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:19.206757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:27:24.574556 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:24.574514    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l" podUID="a2520210-f8cc-47fe-a2b6-dcef3604c80a" containerName="main" containerID="cri-o://50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2" gracePeriod=45
Sep 10 03:27:24.819804 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:24.819778    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:27:25.004768 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.004736    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-model-cache\") pod \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") "
Sep 10 03:27:25.004768 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.004771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-dshm\") pod \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") "
Sep 10 03:27:25.005030 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.004837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tls-certs\") pod \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") "
Sep 10 03:27:25.005030 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.004864    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8dzz\" (UniqueName: \"kubernetes.io/projected/a2520210-f8cc-47fe-a2b6-dcef3604c80a-kube-api-access-x8dzz\") pod \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") "
Sep 10 03:27:25.005030 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.004920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-home\") pod \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") "
Sep 10 03:27:25.005030 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.004937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tmp-dir\") pod \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\" (UID: \"a2520210-f8cc-47fe-a2b6-dcef3604c80a\") "
Sep 10 03:27:25.005276 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.005069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2520210-f8cc-47fe-a2b6-dcef3604c80a" (UID: "a2520210-f8cc-47fe-a2b6-dcef3604c80a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:25.005276 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.005223    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:25.005276 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.005234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-home" (OuterVolumeSpecName: "home") pod "a2520210-f8cc-47fe-a2b6-dcef3604c80a" (UID: "a2520210-f8cc-47fe-a2b6-dcef3604c80a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:25.005420 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.005273    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2520210-f8cc-47fe-a2b6-dcef3604c80a" (UID: "a2520210-f8cc-47fe-a2b6-dcef3604c80a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:25.007264 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.007231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2520210-f8cc-47fe-a2b6-dcef3604c80a" (UID: "a2520210-f8cc-47fe-a2b6-dcef3604c80a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:27:25.007264 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.007243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2520210-f8cc-47fe-a2b6-dcef3604c80a-kube-api-access-x8dzz" (OuterVolumeSpecName: "kube-api-access-x8dzz") pod "a2520210-f8cc-47fe-a2b6-dcef3604c80a" (UID: "a2520210-f8cc-47fe-a2b6-dcef3604c80a"). InnerVolumeSpecName "kube-api-access-x8dzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:27:25.007264 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.007263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-dshm" (OuterVolumeSpecName: "dshm") pod "a2520210-f8cc-47fe-a2b6-dcef3604c80a" (UID: "a2520210-f8cc-47fe-a2b6-dcef3604c80a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:25.106529 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.106494    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8dzz\" (UniqueName: \"kubernetes.io/projected/a2520210-f8cc-47fe-a2b6-dcef3604c80a-kube-api-access-x8dzz\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:25.106529 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.106522    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:25.106529 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.106531    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:25.106529 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.106539    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2520210-f8cc-47fe-a2b6-dcef3604c80a-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:25.106783 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.106547    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2520210-f8cc-47fe-a2b6-dcef3604c80a-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:25.591114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.591021    2580 generic.go:356] "Generic (PLEG): container finished" podID="a2520210-f8cc-47fe-a2b6-dcef3604c80a" containerID="50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2" exitCode=0
Sep 10 03:27:25.591114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.591062    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"
Sep 10 03:27:25.591114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.591071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l" event={"ID":"a2520210-f8cc-47fe-a2b6-dcef3604c80a","Type":"ContainerDied","Data":"50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2"}
Sep 10 03:27:25.591114 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.591110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l" event={"ID":"a2520210-f8cc-47fe-a2b6-dcef3604c80a","Type":"ContainerDied","Data":"a942d413d62f03fe322c9eb665a3c0229333eb51afbd84590c6ab409527d2c1a"}
Sep 10 03:27:25.591743 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.591131    2580 scope.go:117] "RemoveContainer" containerID="50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2"
Sep 10 03:27:25.599908 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.599888    2580 scope.go:117] "RemoveContainer" containerID="50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2"
Sep 10 03:27:25.600175 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:27:25.600149    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2\": container with ID starting with 50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2 not found: ID does not exist" containerID="50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2"
Sep 10 03:27:25.600235 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.600191    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2"} err="failed to get container status \"50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2\": rpc error: code = NotFound desc = could not find container \"50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2\": container with ID starting with 50b90bfd878e74f539753309e02689ff00eb21fd35c0fa9533fae3b5996af8a2 not found: ID does not exist"
Sep 10 03:27:25.608177 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.608144    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"]
Sep 10 03:27:25.611836 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:25.611817    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-082ad071-kserve-796b88b65f-jrt5l"]
Sep 10 03:27:26.267100 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.267052    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg" podUID="72eef7bf-cd1b-4bf8-b2ee-79142510466b" containerName="main" containerID="cri-o://bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25" gracePeriod=45
Sep 10 03:27:26.515137 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.515100    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:27:26.597882 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.597850    2580 generic.go:356] "Generic (PLEG): container finished" podID="72eef7bf-cd1b-4bf8-b2ee-79142510466b" containerID="bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25" exitCode=0
Sep 10 03:27:26.597882 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.597883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"
Sep 10 03:27:26.598361 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.597905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg" event={"ID":"72eef7bf-cd1b-4bf8-b2ee-79142510466b","Type":"ContainerDied","Data":"bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25"}
Sep 10 03:27:26.598361 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.597935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg" event={"ID":"72eef7bf-cd1b-4bf8-b2ee-79142510466b","Type":"ContainerDied","Data":"0c7b728b89b813b7a8313aa98a8eb095791049c17ff0c7355b827a984297e352"}
Sep 10 03:27:26.598361 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.597950    2580 scope.go:117] "RemoveContainer" containerID="bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25"
Sep 10 03:27:26.606235 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.606219    2580 scope.go:117] "RemoveContainer" containerID="bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25"
Sep 10 03:27:26.606462 ip-10-0-142-156 kubenswrapper[2580]: E0910 03:27:26.606446    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25\": container with ID starting with bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25 not found: ID does not exist" containerID="bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25"
Sep 10 03:27:26.606517 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.606469    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25"} err="failed to get container status \"bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25\": rpc error: code = NotFound desc = could not find container \"bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25\": container with ID starting with bc86c06a2a83492798749716fdb348edad880e6c1e7c6e10a2295c308a647f25 not found: ID does not exist"
Sep 10 03:27:26.620846 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.620827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-home\") pod \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") "
Sep 10 03:27:26.620931 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.620917    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tmp-dir\") pod \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") "
Sep 10 03:27:26.620967 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.620945    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-model-cache\") pod \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") "
Sep 10 03:27:26.621001 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.620981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tls-certs\") pod \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") "
Sep 10 03:27:26.621039 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45dlc\" (UniqueName: \"kubernetes.io/projected/72eef7bf-cd1b-4bf8-b2ee-79142510466b-kube-api-access-45dlc\") pod \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") "
Sep 10 03:27:26.621083 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621075    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-dshm\") pod \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\" (UID: \"72eef7bf-cd1b-4bf8-b2ee-79142510466b\") "
Sep 10 03:27:26.621140 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621091    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-home" (OuterVolumeSpecName: "home") pod "72eef7bf-cd1b-4bf8-b2ee-79142510466b" (UID: "72eef7bf-cd1b-4bf8-b2ee-79142510466b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:26.621207 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72eef7bf-cd1b-4bf8-b2ee-79142510466b" (UID: "72eef7bf-cd1b-4bf8-b2ee-79142510466b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:26.621269 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621205    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-model-cache" (OuterVolumeSpecName: "model-cache") pod "72eef7bf-cd1b-4bf8-b2ee-79142510466b" (UID: "72eef7bf-cd1b-4bf8-b2ee-79142510466b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:26.621445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621422    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tmp-dir\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:26.621445 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621446    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-model-cache\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:26.621580 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.621459    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-home\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:26.623069 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.623041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-dshm" (OuterVolumeSpecName: "dshm") pod "72eef7bf-cd1b-4bf8-b2ee-79142510466b" (UID: "72eef7bf-cd1b-4bf8-b2ee-79142510466b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:27:26.623183 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.623119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72eef7bf-cd1b-4bf8-b2ee-79142510466b" (UID: "72eef7bf-cd1b-4bf8-b2ee-79142510466b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:27:26.623240 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.623194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72eef7bf-cd1b-4bf8-b2ee-79142510466b-kube-api-access-45dlc" (OuterVolumeSpecName: "kube-api-access-45dlc") pod "72eef7bf-cd1b-4bf8-b2ee-79142510466b" (UID: "72eef7bf-cd1b-4bf8-b2ee-79142510466b"). InnerVolumeSpecName "kube-api-access-45dlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:27:26.722026 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.721996    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72eef7bf-cd1b-4bf8-b2ee-79142510466b-dshm\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:26.722026 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.722025    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72eef7bf-cd1b-4bf8-b2ee-79142510466b-tls-certs\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:26.722248 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.722038    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45dlc\" (UniqueName: \"kubernetes.io/projected/72eef7bf-cd1b-4bf8-b2ee-79142510466b-kube-api-access-45dlc\") on node \"ip-10-0-142-156.ec2.internal\" DevicePath \"\""
Sep 10 03:27:26.919809 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.919779    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"]
Sep 10 03:27:26.925116 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:26.925090    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-082ad071-kserve-79dcc5c455-jsdtg"]
Sep 10 03:27:27.325128 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:27.325044    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72eef7bf-cd1b-4bf8-b2ee-79142510466b" path="/var/lib/kubelet/pods/72eef7bf-cd1b-4bf8-b2ee-79142510466b/volumes"
Sep 10 03:27:27.325489 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:27.325475    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2520210-f8cc-47fe-a2b6-dcef3604c80a" path="/var/lib/kubelet/pods/a2520210-f8cc-47fe-a2b6-dcef3604c80a/volumes"
Sep 10 03:27:27.686764 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:27.686731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:27:34.810263 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:34.810230    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:27:42.016741 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:42.016705    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:27:43.973215 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:43.973187    2580 scope.go:117] "RemoveContainer" containerID="7356f485b52a45e1d925aef0f5461ad725252b71fbb4e423121dc13ca0a38764"
Sep 10 03:27:50.209330 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:50.209295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:27:58.005320 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:27:58.005275    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t9kpg"]
Sep 10 03:28:07.213623 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:07.213593    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-bk7nj_c5a037c5-f39c-4a10-8113-a3fd352c51d8/copy-manifests/0.log"
Sep 10 03:28:07.220392 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:07.220359    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-bk7nj_c5a037c5-f39c-4a10-8113-a3fd352c51d8/manager/0.log"
Sep 10 03:28:08.007773 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:08.007747    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816820-6mssq_a2464ac4-e3a4-48d7-bdea-7f64e1fc5bef/cleanup/0.log"
Sep 10 03:28:08.119142 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:08.119117    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816835-xphvn_446f3df1-0d84-42a1-b5e0-3158ec89c6f8/cleanup/0.log"
Sep 10 03:28:09.625157 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:09.625125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t9kpg_e6544dd2-ffb2-4e04-8c56-d833b526073d/limitador/0.log"
Sep 10 03:28:10.338990 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:10.338959    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-mpp57_630a69ea-e5b1-473c-b15e-a21790a98a34/kube-auth-proxy/0.log"
Sep 10 03:28:10.567030 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:10.567005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-7z5m6_00b5c9d7-c6ba-46b5-aef5-919ea9f1e780/istio-proxy/1.log"
Sep 10 03:28:11.701816 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:11.701788    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs9vz8_5cb158dd-cb2f-4f14-8189-c500e5ad450d/main/0.log"
Sep 10 03:28:11.921828 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:11.921793    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-g65gp_a586ba56-1ab2-46c2-abab-5a2316319eae/main/0.log"
Sep 10 03:28:17.551730 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.551690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrsp9/must-gather-7k6wp"]
Sep 10 03:28:17.552244 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.552228    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2520210-f8cc-47fe-a2b6-dcef3604c80a" containerName="main"
Sep 10 03:28:17.552291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.552247    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2520210-f8cc-47fe-a2b6-dcef3604c80a" containerName="main"
Sep 10 03:28:17.552291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.552267    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72eef7bf-cd1b-4bf8-b2ee-79142510466b" containerName="main"
Sep 10 03:28:17.552291 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.552275    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72eef7bf-cd1b-4bf8-b2ee-79142510466b" containerName="main"
Sep 10 03:28:17.552386 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.552363    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2520210-f8cc-47fe-a2b6-dcef3604c80a" containerName="main"
Sep 10 03:28:17.552386 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.552379    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72eef7bf-cd1b-4bf8-b2ee-79142510466b" containerName="main"
Sep 10 03:28:17.555522 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.555505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:17.558494 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.558475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrsp9\"/\"openshift-service-ca.crt\""
Sep 10 03:28:17.558823 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.558806    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrsp9\"/\"kube-root-ca.crt\""
Sep 10 03:28:17.560014 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.559999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrsp9\"/\"default-dockercfg-6x58w\""
Sep 10 03:28:17.565490 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.565465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/must-gather-7k6wp"]
Sep 10 03:28:17.571806 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.571783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxl5m\" (UniqueName: \"kubernetes.io/projected/90325b07-16df-44d2-a3d0-febd9bbe2219-kube-api-access-kxl5m\") pod \"must-gather-7k6wp\" (UID: \"90325b07-16df-44d2-a3d0-febd9bbe2219\") " pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:17.571906 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.571845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90325b07-16df-44d2-a3d0-febd9bbe2219-must-gather-output\") pod \"must-gather-7k6wp\" (UID: \"90325b07-16df-44d2-a3d0-febd9bbe2219\") " pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:17.672511 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.672478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90325b07-16df-44d2-a3d0-febd9bbe2219-must-gather-output\") pod \"must-gather-7k6wp\" (UID: \"90325b07-16df-44d2-a3d0-febd9bbe2219\") " pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:17.672686 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.672576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxl5m\" (UniqueName: \"kubernetes.io/projected/90325b07-16df-44d2-a3d0-febd9bbe2219-kube-api-access-kxl5m\") pod \"must-gather-7k6wp\" (UID: \"90325b07-16df-44d2-a3d0-febd9bbe2219\") " pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:17.672841 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.672820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/90325b07-16df-44d2-a3d0-febd9bbe2219-must-gather-output\") pod \"must-gather-7k6wp\" (UID: \"90325b07-16df-44d2-a3d0-febd9bbe2219\") " pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:17.685967 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.685932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxl5m\" (UniqueName: \"kubernetes.io/projected/90325b07-16df-44d2-a3d0-febd9bbe2219-kube-api-access-kxl5m\") pod \"must-gather-7k6wp\" (UID: \"90325b07-16df-44d2-a3d0-febd9bbe2219\") " pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:17.865796 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:17.865769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/must-gather-7k6wp"
Sep 10 03:28:18.138223 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:18.138200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/must-gather-7k6wp"]
Sep 10 03:28:18.140400 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:28:18.140366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90325b07_16df_44d2_a3d0_febd9bbe2219.slice/crio-5e26d4e222c56d92e745e03f5a8a2a0968f7b68dacd9def4b5d78fac85f83d81 WatchSource:0}: Error finding container 5e26d4e222c56d92e745e03f5a8a2a0968f7b68dacd9def4b5d78fac85f83d81: Status 404 returned error can't find the container with id 5e26d4e222c56d92e745e03f5a8a2a0968f7b68dacd9def4b5d78fac85f83d81
Sep 10 03:28:18.142389 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:18.142370    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 03:28:18.848380 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:18.848343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/must-gather-7k6wp" event={"ID":"90325b07-16df-44d2-a3d0-febd9bbe2219","Type":"ContainerStarted","Data":"5e26d4e222c56d92e745e03f5a8a2a0968f7b68dacd9def4b5d78fac85f83d81"}
Sep 10 03:28:19.858375 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:19.858337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/must-gather-7k6wp" event={"ID":"90325b07-16df-44d2-a3d0-febd9bbe2219","Type":"ContainerStarted","Data":"62bec68bab4dc413163ed91105c92212dfa037a8a1b00d8745872585be66f3c9"}
Sep 10 03:28:19.858737 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:19.858384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/must-gather-7k6wp" event={"ID":"90325b07-16df-44d2-a3d0-febd9bbe2219","Type":"ContainerStarted","Data":"898c7f8a2499b49b8acad5f0c50e4298c0f726f5a0a46c06755ab0170cbc4540"}
Sep 10 03:28:19.883526 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:19.883456    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrsp9/must-gather-7k6wp" podStartSLOduration=1.911513099 podStartE2EDuration="2.883434099s" totalImagesPullingTime="971.921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:28:19.881530874 +0000 UTC m=+2497.031430573" watchObservedRunningTime="2026-09-10 03:28:19.883434099 +0000 UTC m=+2497.033333798"
Sep 10 03:28:20.980409 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:20.980382    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rjz9f_610b6131-5edf-4c48-a4d0-240d567c76c3/global-pull-secret-syncer/0.log"
Sep 10 03:28:21.128471 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:21.128439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s28lq_7b4ab09b-8102-437a-8a61-1333d7327dda/konnectivity-agent/0.log"
Sep 10 03:28:21.207525 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:21.207489    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-156.ec2.internal_32f22322537600d0788d094c2580f8d7/haproxy/0.log"
Sep 10 03:28:25.808401 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:25.808321    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t9kpg_e6544dd2-ffb2-4e04-8c56-d833b526073d/limitador/0.log"
Sep 10 03:28:27.443315 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.443286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e99e2739-8452-4ba4-a4e3-3476cb3aa746/alertmanager/0.log"
Sep 10 03:28:27.499728 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.499698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e99e2739-8452-4ba4-a4e3-3476cb3aa746/config-reloader/0.log"
Sep 10 03:28:27.546137 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.546103    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e99e2739-8452-4ba4-a4e3-3476cb3aa746/kube-rbac-proxy-web/0.log"
Sep 10 03:28:27.574983 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.574955    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e99e2739-8452-4ba4-a4e3-3476cb3aa746/kube-rbac-proxy/0.log"
Sep 10 03:28:27.651899 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.651810    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e99e2739-8452-4ba4-a4e3-3476cb3aa746/kube-rbac-proxy-metric/0.log"
Sep 10 03:28:27.700817 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.700782    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e99e2739-8452-4ba4-a4e3-3476cb3aa746/prom-label-proxy/0.log"
Sep 10 03:28:27.742104 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.742070    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e99e2739-8452-4ba4-a4e3-3476cb3aa746/init-config-reloader/0.log"
Sep 10 03:28:27.817739 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.817709    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6fmh5_52a23e89-45ac-4a22-a437-5da898f454d6/kube-state-metrics/0.log"
Sep 10 03:28:27.846106 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.846075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6fmh5_52a23e89-45ac-4a22-a437-5da898f454d6/kube-rbac-proxy-main/0.log"
Sep 10 03:28:27.867848 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:27.867816    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6fmh5_52a23e89-45ac-4a22-a437-5da898f454d6/kube-rbac-proxy-self/0.log"
Sep 10 03:28:28.076403 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.076329    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9svq4_5e5bb6c7-94e4-4ea5-bb1f-34c41b535988/node-exporter/0.log"
Sep 10 03:28:28.106467 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.106434    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9svq4_5e5bb6c7-94e4-4ea5-bb1f-34c41b535988/kube-rbac-proxy/0.log"
Sep 10 03:28:28.139592 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.139562    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9svq4_5e5bb6c7-94e4-4ea5-bb1f-34c41b535988/init-textfile/0.log"
Sep 10 03:28:28.260453 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.260422    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-s77n5_9c53091c-e6fc-48c8-a734-bc27cffe7c7a/kube-rbac-proxy-main/0.log"
Sep 10 03:28:28.298794 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.298759    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-s77n5_9c53091c-e6fc-48c8-a734-bc27cffe7c7a/kube-rbac-proxy-self/0.log"
Sep 10 03:28:28.333028 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.332926    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-s77n5_9c53091c-e6fc-48c8-a734-bc27cffe7c7a/openshift-state-metrics/0.log"
Sep 10 03:28:28.381721 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.381691    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b6cedb2-5f76-4edb-8a15-b87aee32ee5f/prometheus/0.log"
Sep 10 03:28:28.411460 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.411435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b6cedb2-5f76-4edb-8a15-b87aee32ee5f/config-reloader/0.log"
Sep 10 03:28:28.439560 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.439480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b6cedb2-5f76-4edb-8a15-b87aee32ee5f/thanos-sidecar/0.log"
Sep 10 03:28:28.494924 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.494851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b6cedb2-5f76-4edb-8a15-b87aee32ee5f/kube-rbac-proxy-web/0.log"
Sep 10 03:28:28.528427 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.528396    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b6cedb2-5f76-4edb-8a15-b87aee32ee5f/kube-rbac-proxy/0.log"
Sep 10 03:28:28.560876 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.560843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b6cedb2-5f76-4edb-8a15-b87aee32ee5f/kube-rbac-proxy-thanos/0.log"
Sep 10 03:28:28.591370 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.591299    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b6cedb2-5f76-4edb-8a15-b87aee32ee5f/init-config-reloader/0.log"
Sep 10 03:28:28.674254 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.674224    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-294n7_5d2aa94b-ba0c-49d6-8225-ec6daaf68f9d/prometheus-operator-admission-webhook/0.log"
Sep 10 03:28:28.831792 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.831758    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7d6df9c-d22nb_69861a10-4283-4df8-9d4e-70985d88b43b/thanos-query/0.log"
Sep 10 03:28:28.868372 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.868318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7d6df9c-d22nb_69861a10-4283-4df8-9d4e-70985d88b43b/kube-rbac-proxy-web/0.log"
Sep 10 03:28:28.903780 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.903749    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7d6df9c-d22nb_69861a10-4283-4df8-9d4e-70985d88b43b/kube-rbac-proxy/0.log"
Sep 10 03:28:28.938465 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.938436    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7d6df9c-d22nb_69861a10-4283-4df8-9d4e-70985d88b43b/prom-label-proxy/0.log"
Sep 10 03:28:28.971762 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.971736    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7d6df9c-d22nb_69861a10-4283-4df8-9d4e-70985d88b43b/kube-rbac-proxy-rules/0.log"
Sep 10 03:28:28.999252 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:28.999190    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dd7d6df9c-d22nb_69861a10-4283-4df8-9d4e-70985d88b43b/kube-rbac-proxy-metrics/0.log"
Sep 10 03:28:30.260521 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.260480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"]
Sep 10 03:28:30.266059 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.266031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.287344 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.287318    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"]
Sep 10 03:28:30.413042 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.412994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-lib-modules\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.413251 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.413052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-sys\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.413251 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.413102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-proc\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.413251 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.413139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-podres\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.413443 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.413293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fd4x\" (UniqueName: \"kubernetes.io/projected/f9fb447b-eaa7-43e3-bb82-ac3641268996-kube-api-access-4fd4x\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.514484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fd4x\" (UniqueName: \"kubernetes.io/projected/f9fb447b-eaa7-43e3-bb82-ac3641268996-kube-api-access-4fd4x\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.514577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-lib-modules\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.514607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-sys\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.514630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-proc\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.514662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-podres\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.514862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-podres\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.515316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-lib-modules\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515391 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.515369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-sys\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.515838 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.515373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9fb447b-eaa7-43e3-bb82-ac3641268996-proc\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.524508 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.524475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fd4x\" (UniqueName: \"kubernetes.io/projected/f9fb447b-eaa7-43e3-bb82-ac3641268996-kube-api-access-4fd4x\") pod \"perf-node-gather-daemonset-78x5c\" (UID: \"f9fb447b-eaa7-43e3-bb82-ac3641268996\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.577520 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.577490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:30.884177 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.883613    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"]
Sep 10 03:28:30.891258 ip-10-0-142-156 kubenswrapper[2580]: W0910 03:28:30.888022    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9fb447b_eaa7_43e3_bb82_ac3641268996.slice/crio-3f4ac2ed789dc6b7632d17ab97465cecb93ac1b3bd4a55d4ab078e5bcbf69bfa WatchSource:0}: Error finding container 3f4ac2ed789dc6b7632d17ab97465cecb93ac1b3bd4a55d4ab078e5bcbf69bfa: Status 404 returned error can't find the container with id 3f4ac2ed789dc6b7632d17ab97465cecb93ac1b3bd4a55d4ab078e5bcbf69bfa
Sep 10 03:28:30.932858 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:30.932828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c" event={"ID":"f9fb447b-eaa7-43e3-bb82-ac3641268996","Type":"ContainerStarted","Data":"3f4ac2ed789dc6b7632d17ab97465cecb93ac1b3bd4a55d4ab078e5bcbf69bfa"}
Sep 10 03:28:31.939706 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:31.939665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c" event={"ID":"f9fb447b-eaa7-43e3-bb82-ac3641268996","Type":"ContainerStarted","Data":"13dfecf9fc839a8c7e1320d51b63812f5ba2824e93cd9609ecdf1be1346a4bf2"}
Sep 10 03:28:31.940183 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:31.939803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:31.973476 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:31.973423    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c" podStartSLOduration=1.973406528 podStartE2EDuration="1.973406528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 03:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:28:31.972665022 +0000 UTC m=+2509.122564719" watchObservedRunningTime="2026-09-10 03:28:31.973406528 +0000 UTC m=+2509.123306225"
Sep 10 03:28:33.547964 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:33.547939    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9ngr6_6a65c613-798a-4bd4-bd72-e65189642bee/dns/0.log"
Sep 10 03:28:33.571265 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:33.571238    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9ngr6_6a65c613-798a-4bd4-bd72-e65189642bee/kube-rbac-proxy/0.log"
Sep 10 03:28:33.733590 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:33.733560    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6vbtf_f2d9ef21-3103-4690-a438-6e727bd4b91e/dns-node-resolver/0.log"
Sep 10 03:28:34.333318 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:34.333285    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-556d97688d-77wr5_9010e46f-c5b2-4f62-8bf1-d6d6ddd6b305/registry/0.log"
Sep 10 03:28:34.378351 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:34.378323    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzhhr_fc590b11-0eeb-40df-885c-5f5f5b0dd195/node-ca/0.log"
Sep 10 03:28:35.624747 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:35.624706    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-mpp57_630a69ea-e5b1-473c-b15e-a21790a98a34/kube-auth-proxy/0.log"
Sep 10 03:28:35.686809 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:35.686780    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-7z5m6_00b5c9d7-c6ba-46b5-aef5-919ea9f1e780/istio-proxy/0.log"
Sep 10 03:28:35.689908 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:35.689887    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-7z5m6_00b5c9d7-c6ba-46b5-aef5-919ea9f1e780/istio-proxy/1.log"
Sep 10 03:28:36.745084 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:36.745050    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v7vms_55fccc6f-5bd7-47ad-9f36-5eeda877b4e4/serve-healthcheck-canary/0.log"
Sep 10 03:28:37.331877 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:37.331849    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrh84_5aefe13b-20ee-424f-8f13-225eee0253d5/kube-rbac-proxy/0.log"
Sep 10 03:28:37.356494 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:37.356465    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrh84_5aefe13b-20ee-424f-8f13-225eee0253d5/exporter/0.log"
Sep 10 03:28:37.375877 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:37.375850    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrh84_5aefe13b-20ee-424f-8f13-225eee0253d5/extractor/0.log"
Sep 10 03:28:37.957093 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:37.957063    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-78x5c"
Sep 10 03:28:40.833189 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:40.833139    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-bk7nj_c5a037c5-f39c-4a10-8113-a3fd352c51d8/manager/0.log"
Sep 10 03:28:40.855628 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:40.855598    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-bk7nj_c5a037c5-f39c-4a10-8113-a3fd352c51d8/copy-manifests/0.log"
Sep 10 03:28:41.124342 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:41.124307    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-mpp57_630a69ea-e5b1-473c-b15e-a21790a98a34/kube-auth-proxy/0.log"
Sep 10 03:28:41.163866 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:41.163831    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-7z5m6_00b5c9d7-c6ba-46b5-aef5-919ea9f1e780/istio-proxy/0.log"
Sep 10 03:28:41.166079 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:41.166056    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-7z5m6_00b5c9d7-c6ba-46b5-aef5-919ea9f1e780/istio-proxy/1.log"
Sep 10 03:28:42.297608 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:42.297581    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cd97c998c-h65pb_0235f993-41e9-4711-bc40-4a001a538495/manager/0.log"
Sep 10 03:28:43.069843 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:43.069817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-mpp57_630a69ea-e5b1-473c-b15e-a21790a98a34/kube-auth-proxy/0.log"
Sep 10 03:28:43.108134 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:43.108103    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-7z5m6_00b5c9d7-c6ba-46b5-aef5-919ea9f1e780/istio-proxy/0.log"
Sep 10 03:28:43.112406 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:43.112377    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-7z5m6_00b5c9d7-c6ba-46b5-aef5-919ea9f1e780/istio-proxy/1.log"
Sep 10 03:28:46.997511 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:46.997481    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-mr7hd_e86d3abc-da18-4d1d-ae3f-bbe1335c9f10/migrator/0.log"
Sep 10 03:28:47.051667 ip-10-0-142-156 kubenswrapper[2580]: I0910 03:28:47.051600    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-mr7hd_e86d3abc-da18-4d1d-ae3f-bbe1335c9f10/graceful-termination/0.log"