Sep 16 12:01:19.079393 ip-10-0-135-242 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 12:01:19.079403 ip-10-0-135-242 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 12:01:19.079411 ip-10-0-135-242 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 12:01:19.079552 ip-10-0-135-242 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 12:01:29.270808 ip-10-0-135-242 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 12:01:29.270824 ip-10-0-135-242 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ffb7776a74254a7ba2fa2709823b8fe4 --
Sep 16 12:03:22.338215 ip-10-0-135-242 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:03:22.735082 ip-10-0-135-242 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:22.735082 ip-10-0-135-242 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:03:22.735082 ip-10-0-135-242 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:22.735082 ip-10-0-135-242 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:03:22.735612 ip-10-0-135-242 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:22.737446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.737379    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:03:22.740914 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740899    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:22.740914 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740914    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740918    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740921    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740924    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740928    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740931    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740934    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740937    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740951    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740955    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740958    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740964    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740967    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740970    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740973    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740975    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740977    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740980    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740983    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740985    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:22.741003 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740988    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740990    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740993    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740995    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.740998    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741000    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741002    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741005    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741009    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741013    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741016    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741019    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741022    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741025    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741027    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741030    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741033    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741036    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741038    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741041    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:22.741482 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741043    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741046    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741048    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741050    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741053    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741055    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741057    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741061    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741065    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741069    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741071    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741074    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741077    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741080    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741083    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741085    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741088    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741091    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741093    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:22.741982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741096    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741099    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741101    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741104    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741106    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741109    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741111    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741114    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741116    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741119    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741121    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741130    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741134    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741136    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741139    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741141    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741144    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741146    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741149    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:22.742479 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741151    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741154    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741156    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741159    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741161    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741164    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741166    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741170    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741173    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741176    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741180    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741184    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741187    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741190    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741192    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741195    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741198    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741200    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741203    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:22.742937 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741206    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741892    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741899    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741902    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741905    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741909    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741912    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741915    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741917    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741920    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741922    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741925    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741929    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741933    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741936    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741939    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741958    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741961    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741963    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:22.743406 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741966    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741969    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741972    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741974    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741977    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741980    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741982    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741985    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741988    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741991    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741993    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741996    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.741998    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742001    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742004    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742006    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742009    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742011    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742014    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742016    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:22.743879 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742019    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742022    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742024    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742026    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742029    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742031    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742034    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742036    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742038    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742041    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742043    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742047    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742050    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742054    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742056    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742059    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742061    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742065    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742067    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742069    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:22.744389 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742072    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742074    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742077    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742081    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742084    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742087    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742090    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742093    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742096    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742099    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742101    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742103    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742106    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742109    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742113    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742115    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742117    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742120    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742122    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:22.744878 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742125    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742127    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742130    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742132    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742135    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742137    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742140    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742143    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742145    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742148    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742150    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742153    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742155    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742158    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742160    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742162    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742165    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742167    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742169    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742172    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:22.745358 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742174    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742177    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742546    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742551    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742554    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742557    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742561    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742564    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742566    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742569    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742572    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742574    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742576    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742579    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742582    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742584    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742587    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742590    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742595    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:22.745852 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742598    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742600    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742603    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742605    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742608    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742610    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742613    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742615    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742618    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742620    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742623    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742625    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742627    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742630    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742632    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742635    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742638    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742640    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742642    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742648    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:22.746318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742651    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742654    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742656    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742659    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742661    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742663    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742666    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742668    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742671    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742674    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742678    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742683    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742685    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742688    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742691    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742695    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742698    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742701    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742704    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742707    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:22.746809 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742709    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742712    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742714    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742718    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742723    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742725    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742728    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742730    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742733    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742735    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742738    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742741    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742744    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742746    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742749    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742752    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742754    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742756    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742759    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742762    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:22.747351 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742764    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742767    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742769    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742773    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742775    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742777    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742780    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742783    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742785    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742788    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742790    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742793    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742795    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742797    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742800    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742802    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742804    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742807    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742809    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:22.747829 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742812    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742814    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.742817    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742882    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742915    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742920    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742925    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742929    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742932    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742936    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742955    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742958    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742961    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742965    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742968    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742971    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742974    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742978    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742980    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742983    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742986    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742990    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742993    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742996    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:03:22.748314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.742998    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743001    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743005    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743008    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743011    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743014    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743017    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743020    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743022    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743025    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743028    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743032    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743036    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743038    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743041    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743044    2560 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743047    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743050    2560 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743053    2560 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743056    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743059    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743061    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743065    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743067    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743071    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:03:22.748883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743073    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743076    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743079    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743082    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743084    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743087    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743090    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743092    2560 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743096    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743099    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743102    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743104    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743107    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743110    2560 flags.go:64] FLAG: --help="false"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743113    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-135-242.ec2.internal"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743115    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743118    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743121    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743124    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743128    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743131    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743134    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743136    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:03:22.749522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743139    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743142    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743145    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743148    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743150    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743153    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743156    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743158    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743161    2560 flags.go:64] FLAG: --lock-file=""
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743164    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743167    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743169    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743174    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743177    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743180    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743182    2560 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743185    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743188    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743191    2560 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743193    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743197    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743200    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743204    2560 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743206    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743209    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:03:22.750106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743212    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743214    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743217    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743222    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743224    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743231    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743234    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743237    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743240    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743243    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743248    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743251    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743254    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743256    2560 flags.go:64] FLAG: --port="10250"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743259    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743262    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0072aefe248b738c2"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743265    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743268    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743271    2560 flags.go:64] FLAG: --register-node="true"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743273    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743277    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743279    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743282    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743285    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743288    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:03:22.750777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743291    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743294    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743297    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743299    2560 flags.go:64] FLAG: --runonce="false"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743302    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743305    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743308    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743311    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743314    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743316    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743319    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743324    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743327    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743330    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743333    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743335    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743338    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743341    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743344    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743349    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743352    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743354    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743358    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743361    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743363    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:03:22.751391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743366    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743369    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743372    2560 flags.go:64] FLAG: --v="2"
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743375    2560 flags.go:64] FLAG: --version="false"
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743383    2560 flags.go:64] FLAG: --vmodule=""
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743386    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.743389    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743493    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743496    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743499    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743502    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743505    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743508    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743511    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743513    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743516    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743519    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743521    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743526    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743530    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743532    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743535    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:22.752086 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743537    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743540    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743542    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743545    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743548    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743551    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743553    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743555    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743558    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743561    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743563    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743566    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743569    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743571    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743574    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743576    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743579    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743581    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743584    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743586    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:22.752624 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743588    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743591    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743594    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743596    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743598    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743601    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743603    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743606    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743610    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743614    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743616    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743627    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743630    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743633    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743635    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743640    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743644    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743647    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743649    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:22.753154 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743652    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743655    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743657    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743660    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743663    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743666    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743669    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743672    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743675    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743678    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743681    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743683    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743686    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743688    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743691    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743693    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743696    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743698    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743702    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743705    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:22.753635 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743708    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743713    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743717    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743719    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743722    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743726    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743729    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743731    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743734    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743736    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743739    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743742    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743745    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743747    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743750    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743752    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743755    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743757    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743760    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:22.754164 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743762    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743765    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743767    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743770    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743772    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.743775    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.744483    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.751782    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.751796    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.751814    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.751820    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751871    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751876    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751879    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751882    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:22.754677 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751885    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751887    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751890    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751893    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751895    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751899    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751901    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751904    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751908    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751911    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751913    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751915    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751918    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751921    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751923    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751926    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751929    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751932    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751936    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:22.755070 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751954    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751958    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751961    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751964    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751967    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751971    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751975    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751978    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751982    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751985    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751987    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751990    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751993    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751995    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.751998    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752000    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752003    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752005    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752008    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:22.755597 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752011    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752013    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752015    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752018    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752021    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752023    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752026    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752028    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752030    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752033    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752035    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752038    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752040    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752043    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752045    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752048    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752051    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752053    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752056    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752059    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:22.756122 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752061    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752064    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752066    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752069    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752071    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752074    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752077    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752079    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752082    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752084    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752087    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752089    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752092    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752095    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752097    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752100    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752103    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752106    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752108    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752111    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:22.756649 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752113    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752116    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752120    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752122    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752125    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752127    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752130    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752132    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752135    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752138    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752140    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752144    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752146    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752149    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752152    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752154    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:22.757313 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752156    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.752161    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752263    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752268    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752271    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752274    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752277    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752279    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752282    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752285    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752287    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752290    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752292    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752295    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752298    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:22.757713 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752300    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752303    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752305    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752308    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752312    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752316    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752319    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752322    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752325    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752328    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752330    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752333    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752337    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752339    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752342    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752344    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752347    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752350    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752352    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752355    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:22.758149 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752358    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752361    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752364    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752366    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752369    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752372    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752375    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752378    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752380    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752383    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752385    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752388    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752391    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752393    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752396    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752398    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752401    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752403    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752405    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752408    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:22.758828 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752410    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752413    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752415    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752418    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752420    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752423    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752425    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752428    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752430    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752433    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752435    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752437    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752440    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752443    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752445    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752448    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752450    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752452    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752455    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:22.759536 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752458    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752461    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752464    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752466    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752468    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752471    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752474    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752476    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752479    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752481    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752484    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752486    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752488    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752491    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752493    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752496    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752498    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752501    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752503    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752506    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:22.760057 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752508    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752512    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752515    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752518    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752521    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752524    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:22.752526    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.752531    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.752621    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.754414    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.755225    2560 server.go:1019] "Starting client certificate rotation"
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.755339    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.756074    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:03:22.760627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.757772    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:03:22.761053 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.760660    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:03:22.769104 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.769089    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:03:22.772122 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.772106    2560 log.go:25] "Validated CRI v1 image API"
Sep 16 12:03:22.773003 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.772986    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:03:22.775908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.775887    2560 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e3178415-d181-47de-b6d1-f906a6787450:/dev/nvme0n1p3 ec5f3cc7-2f1a-4173-863b-893dc9f57958:/dev/nvme0n1p4]
Sep 16 12:03:22.775986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.775908    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:03:22.781747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.781640    2560 manager.go:216] Machine: {Timestamp:2026-09-16 12:03:22.780301461 +0000 UTC m=+0.329465604 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099827 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25547c4d782e945de9ddc5f8613b70 SystemUUID:ec25547c-4d78-2e94-5de9-ddc5f8613b70 BootID:ffb7776a-7425-4a7b-a2fa-2709823b8fe4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:b1:ff:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:b1:ff:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:cc:3d:54:7c:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:03:22.781747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.781744    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:03:22.781852 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.781809    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:03:22.785225 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.785204    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:03:22.785364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.785228    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:03:22.786134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.786121    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:03:22.786174 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.786139    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:03:22.786174 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.786148    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:03:22.786706 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.786696    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:03:22.787387 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.787378    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:03:22.788218 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.788203    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:03:22.791697 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.791685    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:03:22.791815 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.791806    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:03:22.791846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.791836    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:03:22.791874 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.791859    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:03:22.791923 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.791883    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:03:22.791923 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.791896    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:03:22.791923 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.791909    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:03:22.792262 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.792251    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:03:22.792294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.792270    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:03:22.794360 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.794344    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:03:22.796581 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.796568    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:03:22.796644 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.796597    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:03:22.799503 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799488    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:03:22.799503 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799507    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799513    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799518    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799526    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799532    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799545    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799549    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799556    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799561    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799570    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:03:22.799617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.799574    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:03:22.800157 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.800147    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:03:22.800157 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.800156    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:03:22.800990 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.800969    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:03:22.800990 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.800970    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:03:22.803621 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.803600    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:03:22.804263 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.804249    2560 server.go:1268] "Started kubelet"
Sep 16 12:03:22.804325 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.804291    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:03:22.804461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.804388    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:03:22.804522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.804488    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:03:22.804931 ip-10-0-135-242 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:03:22.808656 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.808633    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:03:22.809669 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.809646    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:03:22.810168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.810153    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:03:22.812281 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.811420    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-242.ec2.internal.18d5cb85e0504789  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-242.ec2.internal,UID:ip-10-0-135-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-242.ec2.internal,},FirstTimestamp:2026-09-16 12:03:22.804086665 +0000 UTC m=+0.353250809,LastTimestamp:2026-09-16 12:03:22.804086665 +0000 UTC m=+0.353250809,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-242.ec2.internal,}"
Sep 16 12:03:22.812558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.812544    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:03:22.812977 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.812963    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:03:22.813132 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.813110    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:22.813485 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.813466    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:03:22.813562 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.813498    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:03:22.813562 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.813514    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:03:22.813668 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.813604    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:03:22.813668 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.813615    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:03:22.813763 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.813744    2560 factory.go:55] Registering systemd factory
Sep 16 12:03:22.813763 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.813766    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:03:22.814015 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.814001    2560 factory.go:153] Registering CRI-O factory
Sep 16 12:03:22.814015 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.814016    2560 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:03:22.814133 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.814064    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:03:22.814133 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.814078    2560 factory.go:103] Registering Raw factory
Sep 16 12:03:22.814133 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.814088    2560 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:03:22.814455 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.814338    2560 manager.go:349] Starting recovery of all containers
Sep 16 12:03:22.819149 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.819123    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wdclr"
Sep 16 12:03:22.819407 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.819337    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:03:22.819470 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.819442    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:03:22.824760 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.824748    2560 manager.go:354] Recovery completed
Sep 16 12:03:22.826068 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.825932    2560 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 12:03:22.828751 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.828725    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:22.828847 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.828825    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wdclr"
Sep 16 12:03:22.830616 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.830581    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:22.830616 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.830615    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:22.830616 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.830630    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:22.831258 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.831244    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:03:22.831258 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.831257    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:03:22.831357 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.831284    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:03:22.831357 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.831317    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:22.832038 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.832020    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:22.832097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.832044    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:22.832097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.832053    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:22.832350 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.832267    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-242.ec2.internal.18d5cb85e1e4c933  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-242.ec2.internal,UID:ip-10-0-135-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-242.ec2.internal,},FirstTimestamp:2026-09-16 12:03:22.830596403 +0000 UTC m=+0.379760548,LastTimestamp:2026-09-16 12:03:22.830596403 +0000 UTC m=+0.379760548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-242.ec2.internal,}"
Sep 16 12:03:22.833269 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.833258    2560 policy_none.go:49] "None policy: Start"
Sep 16 12:03:22.833360 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.833274    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:03:22.833360 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.833283    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:03:22.835237 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.835226    2560 policy_none.go:47] "Start"
Sep 16 12:03:22.869617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.869494    2560 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.869644    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.869657    2560 server.go:87] "Starting device plugin registration server"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.869848    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.869857    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.869968    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.870057    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.870066    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.870498    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:03:22.886446 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.870547    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:22.902721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.902692    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:03:22.904003 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.903986    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:03:22.904101 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.904030    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:03:22.904101 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.904047    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:03:22.904101 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.904084    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:03:22.906214 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.906197    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:22.970223 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.970200    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:22.971025 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.971003    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:22.971101 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.971034    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:22.971101 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.971045    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:22.971101 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.971066    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-242.ec2.internal"
Sep 16 12:03:22.981669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:22.981649    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-242.ec2.internal"
Sep 16 12:03:22.981669 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.981668    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-242.ec2.internal\": node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:22.997360 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:22.997309    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.004352 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.004332    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"]
Sep 16 12:03:23.004409 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.004393    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.005099 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.005085    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.005188 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.005109    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.005188 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.005119    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.007412 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.007398    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.007545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.007531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.007604 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.007565    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.008084 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.008069    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.008162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.008069    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.008162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.008120    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.008162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.008137    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.008162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.008094    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.008290 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.008168    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.010404 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.010389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.010449 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.010418    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:23.015040 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.015018    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:23.015110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.015044    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:23.015110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.015062    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:23.027506 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.027485    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-242.ec2.internal\" not found" node="ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.033131 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.033114    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-242.ec2.internal\" not found" node="ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.097919 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.097900    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.115379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.115357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc20f8ec2b2204fa94263ebc59d562b0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"cc20f8ec2b2204fa94263ebc59d562b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.115457 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.115387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1a289088fc81807ef16ec01ed9b503-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"cf1a289088fc81807ef16ec01ed9b503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.115457 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.115414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1a289088fc81807ef16ec01ed9b503-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"cf1a289088fc81807ef16ec01ed9b503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.198493 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.198473    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.215875 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.215854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc20f8ec2b2204fa94263ebc59d562b0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"cc20f8ec2b2204fa94263ebc59d562b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.215875 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.215862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc20f8ec2b2204fa94263ebc59d562b0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"cc20f8ec2b2204fa94263ebc59d562b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.216030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.215902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1a289088fc81807ef16ec01ed9b503-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"cf1a289088fc81807ef16ec01ed9b503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.216030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.215918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1a289088fc81807ef16ec01ed9b503-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"cf1a289088fc81807ef16ec01ed9b503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.216030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.215964    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf1a289088fc81807ef16ec01ed9b503-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"cf1a289088fc81807ef16ec01ed9b503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.216030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.216015    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf1a289088fc81807ef16ec01ed9b503-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"cf1a289088fc81807ef16ec01ed9b503\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.299263 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.299211    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.329722 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.329699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.335507 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.335490    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.399898 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.399875    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.500420 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.500385    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.600930 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.600869    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.639559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.639535    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:23.701467 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.701437    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 16 12:03:23.754920 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.754880    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:03:23.755358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.755037    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:03:23.755358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.755103    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:03:23.779327 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.779305    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:23.792795 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.792775    2560 apiserver.go:52] "Watching apiserver"
Sep 16 12:03:23.801734 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.801711    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:03:23.802646 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.802617    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pvbqc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr","openshift-cluster-node-tuning-operator/tuned-tgsg7","openshift-dns/node-resolver-7lbrw","openshift-image-registry/node-ca-tbjcp","openshift-multus/multus-additional-cni-plugins-45fxl","openshift-network-operator/iptables-alerter-vsmqm","openshift-ovn-kubernetes/ovnkube-node-jw5g2","kube-system/konnectivity-agent-lzfbc","openshift-multus/multus-5dktj","openshift-multus/network-metrics-daemon-sdmtp","openshift-network-diagnostics/network-check-target-q6vwr"]
Sep 16 12:03:23.805615 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.805596    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.805719 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.805695    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:23.808163 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.808142    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.810837 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.810809    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:03:23.810962 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.810846    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:03:23.811030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.811003    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:03:23.811301 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.811281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7k52n\""
Sep 16 12:03:23.811390 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.811343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.812875 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.812854    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:03:23.813307 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.813285    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.813938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.813915    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:03:23.814205 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.814183    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fmwkh\""
Sep 16 12:03:23.814400 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.814369    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:03:23.814505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.814470    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.816987 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.816968    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9z6zp\""
Sep 16 12:03:23.816987 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.816981    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:03:23.817148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.817026    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:23.817239 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.817219    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:03:23.819435 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.819435 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-var-lib-kubelet\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-host\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-tuned\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-socket-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.819584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-tmp\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4698db6-e801-4511-8142-6a53c3338306-kubelet-config\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.819584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4698db6-e801-4511-8142-6a53c3338306-dbus\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.819761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysconfig\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysctl-d\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-systemd\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-sys\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-sys-fs\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.819761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysctl-conf\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-registration-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-device-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819866    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9rr\" (UniqueName: \"kubernetes.io/projected/b5959c78-448e-4322-9535-fbfd4b5efa97-kube-api-access-jr9rr\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-modprobe-d\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-kubernetes\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-run\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.819996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.819985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnjcq\" (UniqueName: \"kubernetes.io/projected/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-kube-api-access-tnjcq\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.820342 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.820009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9781e9-d23e-44bb-a141-1fa6790be59b-tmp-dir\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.820342 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.820034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.820342 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.820056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-lib-modules\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.820342 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.820093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec9781e9-d23e-44bb-a141-1fa6790be59b-hosts-file\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.820342 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.820127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87gw\" (UniqueName: \"kubernetes.io/projected/ec9781e9-d23e-44bb-a141-1fa6790be59b-kube-api-access-g87gw\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.822188 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.822172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:23.823469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.823446    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:03:23.823708 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.823648    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:03:23.823805 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.823710    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:03:23.823805 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.823721    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zm2xc\""
Sep 16 12:03:23.823805 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.823740    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:03:23.823923 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.823815    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:03:23.824171 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.824153    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:03:23.824711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.824337    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:03:23.824711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.824443    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:03:23.824711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.824451    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xvwkp\""
Sep 16 12:03:23.824711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.824451    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:03:23.824909 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.824876    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.825167 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.825144    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tkxmv\""
Sep 16 12:03:23.825524 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.825505    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:03:23.825601 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.825531    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:03:23.827341 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.827325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:23.828848 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.828827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:03:23.829016 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829002    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:03:23.829109 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829094    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:03:23.829149 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829136    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xfx8j\""
Sep 16 12:03:23.829186 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829163    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:03:23.829223 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829194    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:03:23.829268 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829166    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:03:23.829328 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829312    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 16 12:03:23.829388 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829355    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:03:23.829821 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829803    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:03:23.829916 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829830    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:03:23.829989 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.829962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.830296 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.830281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7xhgt\""
Sep 16 12:03:23.832158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.832112    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 11:58:24 +0000 UTC" deadline="2028-05-04 23:34:34.840764948 +0000 UTC"
Sep 16 12:03:23.832158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.832157    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14315h31m11.008611703s"
Sep 16 12:03:23.832689 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.832665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:23.832834 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.832725    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:23.832834 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.832664    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jchx5\""
Sep 16 12:03:23.832936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.832874    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:03:23.833303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.833278    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:03:23.835594 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.835572    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"]
Sep 16 12:03:23.835700 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.835650    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:23.835757 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.835702    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:23.838783 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.838764    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:03:23.838856 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.838816    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"]
Sep 16 12:03:23.914529 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.914508    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:03:23.920611 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-registration-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.920716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-cnibin\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.920716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-conf-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.920716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:23.920716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-device-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.920716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-modprobe-d\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-registration-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-kubernetes\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-device-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-kubernetes\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-run\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkns\" (UniqueName: \"kubernetes.io/projected/205eab7e-100c-46e7-b39b-e3aef2be4adf-kube-api-access-2bkns\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-modprobe-d\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovnkube-config\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-run\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87gw\" (UniqueName: \"kubernetes.io/projected/ec9781e9-d23e-44bb-a141-1fa6790be59b-kube-api-access-g87gw\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-cni-multus\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-daemon-config\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-multus-certs\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.920984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.920997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovnkube-script-lib\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-host\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-cni-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-etc-kubernetes\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-host\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nl6\" (UniqueName: \"kubernetes.io/projected/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-kube-api-access-k4nl6\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-systemd-units\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-slash\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-socket-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qnx9\" (UniqueName: \"kubernetes.io/projected/874c0c70-1ccb-43b6-b102-fa109d487cc5-kube-api-access-7qnx9\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921335    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4698db6-e801-4511-8142-6a53c3338306-dbus\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-socket-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysconfig\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysctl-d\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.921469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921414    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysconfig\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-systemd\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-system-cni-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4698db6-e801-4511-8142-6a53c3338306-dbus\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-systemd\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-netns\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-cni-bin\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysctl-d\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-sys-fs\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-iptables-alerter-script\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-host-slash\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/205eab7e-100c-46e7-b39b-e3aef2be4adf-serviceca\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovn-node-metrics-cert\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45b8\" (UniqueName: \"kubernetes.io/projected/3ceb08b2-e885-41d1-8e15-7b66decdc24a-kube-api-access-p45b8\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.922005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/205eab7e-100c-46e7-b39b-e3aef2be4adf-host\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-systemd\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-sys-fs\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.921727    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-cni-bin\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:23.921800    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret podName:a4698db6-e801-4511-8142-6a53c3338306 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:24.421779412 +0000 UTC m=+1.970943549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret") pod "global-pull-secret-syncer-pvbqc" (UID: "a4698db6-e801-4511-8142-6a53c3338306") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9rr\" (UniqueName: \"kubernetes.io/projected/b5959c78-448e-4322-9535-fbfd4b5efa97-kube-api-access-jr9rr\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnjcq\" (UniqueName: \"kubernetes.io/projected/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-kube-api-access-tnjcq\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9781e9-d23e-44bb-a141-1fa6790be59b-tmp-dir\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-hostroot\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-kubelet\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-run-netns\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-var-lib-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.921999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-node-log\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-lib-modules\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.922545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec9781e9-d23e-44bb-a141-1fa6790be59b-hosts-file\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk56r\" (UniqueName: \"kubernetes.io/projected/3360bde0-4498-4c93-85ca-0813afa0fc95-kube-api-access-rk56r\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-cni-netd\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ec9781e9-d23e-44bb-a141-1fa6790be59b-hosts-file\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-lib-modules\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9781e9-d23e-44bb-a141-1fa6790be59b-tmp-dir\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5959c78-448e-4322-9535-fbfd4b5efa97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-var-lib-kubelet\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-tuned\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ceb08b2-e885-41d1-8e15-7b66decdc24a-cni-binary-copy\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-ovn\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-var-lib-kubelet\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-env-overrides\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-tmp\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-socket-dir-parent\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.923143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-kubelet\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-cnibin\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4698db6-e801-4511-8142-6a53c3338306-kubelet-config\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d95554cf-2a64-4e58-8e24-275ebf9ffb78-agent-certs\") pod \"konnectivity-agent-lzfbc\" (UID: \"d95554cf-2a64-4e58-8e24-275ebf9ffb78\") " pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d95554cf-2a64-4e58-8e24-275ebf9ffb78-konnectivity-ca\") pod \"konnectivity-agent-lzfbc\" (UID: \"d95554cf-2a64-4e58-8e24-275ebf9ffb78\") " pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqsw\" (UniqueName: \"kubernetes.io/projected/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-kube-api-access-9tqsw\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4698db6-e801-4511-8142-6a53c3338306-kubelet-config\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-sys\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-sys\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-k8s-cni-cncf-io\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922603    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-log-socket\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysctl-conf\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-etc-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:23.923634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.922846    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-sysctl-conf\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.925550 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.925535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-etc-tuned\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.925593 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.925576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-tmp\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.930260 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.930234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnjcq\" (UniqueName: \"kubernetes.io/projected/9a38493c-c3f2-4b9e-8ac4-85bad907ee14-kube-api-access-tnjcq\") pod \"tuned-tgsg7\" (UID: \"9a38493c-c3f2-4b9e-8ac4-85bad907ee14\") " pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:23.930482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.930452    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87gw\" (UniqueName: \"kubernetes.io/projected/ec9781e9-d23e-44bb-a141-1fa6790be59b-kube-api-access-g87gw\") pod \"node-resolver-7lbrw\" (UID: \"ec9781e9-d23e-44bb-a141-1fa6790be59b\") " pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:23.930793 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.930775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9rr\" (UniqueName: \"kubernetes.io/projected/b5959c78-448e-4322-9535-fbfd4b5efa97-kube-api-access-jr9rr\") pod \"aws-ebs-csi-driver-node-gkmgr\" (UID: \"b5959c78-448e-4322-9535-fbfd4b5efa97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:23.935758 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.935738    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m52bm"
Sep 16 12:03:23.942993 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.942974    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m52bm"
Sep 16 12:03:23.976605 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:23.976580    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1a289088fc81807ef16ec01ed9b503.slice/crio-4bb3cd8ca24752f06e262e51d8a34276e79fb2bbe642e49699863d2ff740e431 WatchSource:0}: Error finding container 4bb3cd8ca24752f06e262e51d8a34276e79fb2bbe642e49699863d2ff740e431: Status 404 returned error can't find the container with id 4bb3cd8ca24752f06e262e51d8a34276e79fb2bbe642e49699863d2ff740e431
Sep 16 12:03:23.976819 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:23.976800    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc20f8ec2b2204fa94263ebc59d562b0.slice/crio-9a74e86b51e4b14f19545ff6d57ff6dfce186d6663f3625d820ac2b124cbb4a1 WatchSource:0}: Error finding container 9a74e86b51e4b14f19545ff6d57ff6dfce186d6663f3625d820ac2b124cbb4a1: Status 404 returned error can't find the container with id 9a74e86b51e4b14f19545ff6d57ff6dfce186d6663f3625d820ac2b124cbb4a1
Sep 16 12:03:23.980367 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:23.980353    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:03:24.023584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk56r\" (UniqueName: \"kubernetes.io/projected/3360bde0-4498-4c93-85ca-0813afa0fc95-kube-api-access-rk56r\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:24.023584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-cni-netd\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ceb08b2-e885-41d1-8e15-7b66decdc24a-cni-binary-copy\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-ovn\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-env-overrides\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-socket-dir-parent\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-kubelet\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-cni-netd\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023690    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-kubelet\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-ovn\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-socket-dir-parent\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.023740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-cnibin\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d95554cf-2a64-4e58-8e24-275ebf9ffb78-agent-certs\") pod \"konnectivity-agent-lzfbc\" (UID: \"d95554cf-2a64-4e58-8e24-275ebf9ffb78\") " pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d95554cf-2a64-4e58-8e24-275ebf9ffb78-konnectivity-ca\") pod \"konnectivity-agent-lzfbc\" (UID: \"d95554cf-2a64-4e58-8e24-275ebf9ffb78\") " pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-cnibin\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqsw\" (UniqueName: \"kubernetes.io/projected/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-kube-api-access-9tqsw\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-k8s-cni-cncf-io\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-log-socket\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-etc-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-cnibin\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.023994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-etc-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-conf-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-log-socket\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.024264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkns\" (UniqueName: \"kubernetes.io/projected/205eab7e-100c-46e7-b39b-e3aef2be4adf-kube-api-access-2bkns\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovnkube-config\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-cni-multus\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024162    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-conf-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-daemon-config\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024189    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-env-overrides\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-multus-certs\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024244    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-cni-multus\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-k8s-cni-cncf-io\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ceb08b2-e885-41d1-8e15-7b66decdc24a-cni-binary-copy\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-multus-certs\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-cnibin\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d95554cf-2a64-4e58-8e24-275ebf9ffb78-konnectivity-ca\") pod \"konnectivity-agent-lzfbc\" (UID: \"d95554cf-2a64-4e58-8e24-275ebf9ffb78\") " pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovnkube-script-lib\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-cni-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025059 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-etc-kubernetes\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nl6\" (UniqueName: \"kubernetes.io/projected/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-kube-api-access-k4nl6\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-systemd-units\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-slash\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024585    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-cni-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qnx9\" (UniqueName: \"kubernetes.io/projected/874c0c70-1ccb-43b6-b102-fa109d487cc5-kube-api-access-7qnx9\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-system-cni-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024697    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-netns\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-cni-bin\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-etc-kubernetes\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-iptables-alerter-script\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-host-slash\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovnkube-config\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.025844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/205eab7e-100c-46e7-b39b-e3aef2be4adf-serviceca\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovn-node-metrics-cert\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45b8\" (UniqueName: \"kubernetes.io/projected/3ceb08b2-e885-41d1-8e15-7b66decdc24a-kube-api-access-p45b8\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ceb08b2-e885-41d1-8e15-7b66decdc24a-multus-daemon-config\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.024976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/205eab7e-100c-46e7-b39b-e3aef2be4adf-host\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-systemd\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-cni-bin\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-var-lib-cni-bin\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-hostroot\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-kubelet\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovnkube-script-lib\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-system-cni-dir\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-run-netns\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-var-lib-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-systemd-units\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-node-log\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-host-run-netns\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.026379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025244    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-run-systemd\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-slash\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-cni-bin\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-host-slash\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ceb08b2-e885-41d1-8e15-7b66decdc24a-hostroot\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/205eab7e-100c-46e7-b39b-e3aef2be4adf-serviceca\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-iptables-alerter-script\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-node-log\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.025440    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-var-lib-openvswitch\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/874c0c70-1ccb-43b6-b102-fa109d487cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025563    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-kubelet\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/205eab7e-100c-46e7-b39b-e3aef2be4adf-host\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-host-run-netns\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.026911 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.025602    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:24.525585767 +0000 UTC m=+2.074749899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:24.027416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.025621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/874c0c70-1ccb-43b6-b102-fa109d487cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.027416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.026361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d95554cf-2a64-4e58-8e24-275ebf9ffb78-agent-certs\") pod \"konnectivity-agent-lzfbc\" (UID: \"d95554cf-2a64-4e58-8e24-275ebf9ffb78\") " pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:24.027416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.027308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-ovn-node-metrics-cert\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.030303 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.030285    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:24.030303 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.030303    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:24.030434 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.030312    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hmsnn for pod openshift-network-diagnostics/network-check-target-q6vwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:24.030434 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.030359    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn podName:e4e1f017-a546-4cf7-829d-10ab78aba646 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:24.530346161 +0000 UTC m=+2.079510312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hmsnn" (UniqueName: "kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn") pod "network-check-target-q6vwr" (UID: "e4e1f017-a546-4cf7-829d-10ab78aba646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:24.032230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.032210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkns\" (UniqueName: \"kubernetes.io/projected/205eab7e-100c-46e7-b39b-e3aef2be4adf-kube-api-access-2bkns\") pod \"node-ca-tbjcp\" (UID: \"205eab7e-100c-46e7-b39b-e3aef2be4adf\") " pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:24.032535 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.032517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqsw\" (UniqueName: \"kubernetes.io/projected/a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6-kube-api-access-9tqsw\") pod \"ovnkube-node-jw5g2\" (UID: \"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.032582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.032556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk56r\" (UniqueName: \"kubernetes.io/projected/3360bde0-4498-4c93-85ca-0813afa0fc95-kube-api-access-rk56r\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:24.035973 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.035939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45b8\" (UniqueName: \"kubernetes.io/projected/3ceb08b2-e885-41d1-8e15-7b66decdc24a-kube-api-access-p45b8\") pod \"multus-5dktj\" (UID: \"3ceb08b2-e885-41d1-8e15-7b66decdc24a\") " pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.038852 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.038833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qnx9\" (UniqueName: \"kubernetes.io/projected/874c0c70-1ccb-43b6-b102-fa109d487cc5-kube-api-access-7qnx9\") pod \"multus-additional-cni-plugins-45fxl\" (UID: \"874c0c70-1ccb-43b6-b102-fa109d487cc5\") " pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.039014 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.038998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nl6\" (UniqueName: \"kubernetes.io/projected/92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0-kube-api-access-k4nl6\") pod \"iptables-alerter-vsmqm\" (UID: \"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0\") " pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:24.134699 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.134652    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr"
Sep 16 12:03:24.143654 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.143636    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5959c78_448e_4322_9535_fbfd4b5efa97.slice/crio-68642a23cb6f35c664e76681db1e578e0f25522dfb2bfb4a8702d3b8ac3bbdab WatchSource:0}: Error finding container 68642a23cb6f35c664e76681db1e578e0f25522dfb2bfb4a8702d3b8ac3bbdab: Status 404 returned error can't find the container with id 68642a23cb6f35c664e76681db1e578e0f25522dfb2bfb4a8702d3b8ac3bbdab
Sep 16 12:03:24.152608 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.152592    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tgsg7"
Sep 16 12:03:24.157365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.157349    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7lbrw"
Sep 16 12:03:24.160140 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.160011    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a38493c_c3f2_4b9e_8ac4_85bad907ee14.slice/crio-3872f5c2b91746ed58b9e8e0fd86abcc4bbcd34895100cfb8fe5845dcc5789ba WatchSource:0}: Error finding container 3872f5c2b91746ed58b9e8e0fd86abcc4bbcd34895100cfb8fe5845dcc5789ba: Status 404 returned error can't find the container with id 3872f5c2b91746ed58b9e8e0fd86abcc4bbcd34895100cfb8fe5845dcc5789ba
Sep 16 12:03:24.166502 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.166483    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9781e9_d23e_44bb_a141_1fa6790be59b.slice/crio-70fde5b432085f8d592ffc79b13edb0b8754f9eff5e7243ded42ec181f1c4c94 WatchSource:0}: Error finding container 70fde5b432085f8d592ffc79b13edb0b8754f9eff5e7243ded42ec181f1c4c94: Status 404 returned error can't find the container with id 70fde5b432085f8d592ffc79b13edb0b8754f9eff5e7243ded42ec181f1c4c94
Sep 16 12:03:24.171780 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.171764    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbjcp"
Sep 16 12:03:24.180239 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.180218    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205eab7e_100c_46e7_b39b_e3aef2be4adf.slice/crio-c17e7f786a275a83d424e75cccc346057d68db67664aa1a1ef7c914a1efe7f59 WatchSource:0}: Error finding container c17e7f786a275a83d424e75cccc346057d68db67664aa1a1ef7c914a1efe7f59: Status 404 returned error can't find the container with id c17e7f786a275a83d424e75cccc346057d68db67664aa1a1ef7c914a1efe7f59
Sep 16 12:03:24.187299 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.187275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-45fxl"
Sep 16 12:03:24.192967 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.192934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vsmqm"
Sep 16 12:03:24.194520 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.194501    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874c0c70_1ccb_43b6_b102_fa109d487cc5.slice/crio-4dc2fd6efce3e053f8648b4ba69a91308fd4fc61827341f963efbcadb63e61d3 WatchSource:0}: Error finding container 4dc2fd6efce3e053f8648b4ba69a91308fd4fc61827341f963efbcadb63e61d3: Status 404 returned error can't find the container with id 4dc2fd6efce3e053f8648b4ba69a91308fd4fc61827341f963efbcadb63e61d3
Sep 16 12:03:24.200456 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.200437    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92db4e9b_0f8f_4dd1_a3f8_bbe9d2ef59a0.slice/crio-ef59a8fb8184de92f02c76a7ca21ba75dcddf03360aa55bdd5a85057c68885db WatchSource:0}: Error finding container ef59a8fb8184de92f02c76a7ca21ba75dcddf03360aa55bdd5a85057c68885db: Status 404 returned error can't find the container with id ef59a8fb8184de92f02c76a7ca21ba75dcddf03360aa55bdd5a85057c68885db
Sep 16 12:03:24.207137 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.207122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:24.215047 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.215030    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50f42ac_142a_4dbc_93d0_b8a9a56a9ed6.slice/crio-b696e363a3ff242b7722d2f5ae9c04aa44e193c899b7dd300dc5eda88d2cb11a WatchSource:0}: Error finding container b696e363a3ff242b7722d2f5ae9c04aa44e193c899b7dd300dc5eda88d2cb11a: Status 404 returned error can't find the container with id b696e363a3ff242b7722d2f5ae9c04aa44e193c899b7dd300dc5eda88d2cb11a
Sep 16 12:03:24.236589 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.236556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:24.241247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.241225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dktj"
Sep 16 12:03:24.244012 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.243991    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95554cf_2a64_4e58_8e24_275ebf9ffb78.slice/crio-33dd9b12dcf4a85264b84d44f451375fb914098f67901f2125dc1b97e5a522b7 WatchSource:0}: Error finding container 33dd9b12dcf4a85264b84d44f451375fb914098f67901f2125dc1b97e5a522b7: Status 404 returned error can't find the container with id 33dd9b12dcf4a85264b84d44f451375fb914098f67901f2125dc1b97e5a522b7
Sep 16 12:03:24.250203 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:24.250187    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceb08b2_e885_41d1_8e15_7b66decdc24a.slice/crio-c06de98b6d360598b1a047cd103600079cd9961558ea3d57b8e68067bf669502 WatchSource:0}: Error finding container c06de98b6d360598b1a047cd103600079cd9961558ea3d57b8e68067bf669502: Status 404 returned error can't find the container with id c06de98b6d360598b1a047cd103600079cd9961558ea3d57b8e68067bf669502
Sep 16 12:03:24.265391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.265375    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:24.427662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:24.427600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:24.427791 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.427761    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:24.427846 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:24.427833    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret podName:a4698db6-e801-4511-8142-6a53c3338306 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:25.42781184 +0000 UTC m=+2.976975985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret") pod "global-pull-secret-syncer-pvbqc" (UID: "a4698db6-e801-4511-8142-6a53c3338306") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:25.942118 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:25.942095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:25.942224 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:25.942203    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:25.942276 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:25.942240    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:26.942228592 +0000 UTC m=+3.077550554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:26.043298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.043267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:26.043496 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.043481    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:26.043559 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.043505    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:26.043559 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.043518    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hmsnn for pod openshift-network-diagnostics/network-check-target-q6vwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:26.043661 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.043628    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn podName:e4e1f017-a546-4cf7-829d-10ab78aba646 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:27.043608186 +0000 UTC m=+3.178930152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmsnn" (UniqueName: "kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn") pod "network-check-target-q6vwr" (UID: "e4e1f017-a546-4cf7-829d-10ab78aba646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:26.321333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.321257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:26.322286 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.321395    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:26.322286 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.321813    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:26.322286 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.321901    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:26.344666 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.344610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vsmqm" event={"ID":"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0","Type":"ContainerStarted","Data":"ef59a8fb8184de92f02c76a7ca21ba75dcddf03360aa55bdd5a85057c68885db"}
Sep 16 12:03:26.349478 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.349449    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerStarted","Data":"4dc2fd6efce3e053f8648b4ba69a91308fd4fc61827341f963efbcadb63e61d3"}
Sep 16 12:03:26.354986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.354962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbjcp" event={"ID":"205eab7e-100c-46e7-b39b-e3aef2be4adf","Type":"ContainerStarted","Data":"c17e7f786a275a83d424e75cccc346057d68db67664aa1a1ef7c914a1efe7f59"}
Sep 16 12:03:26.358316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.358291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr" event={"ID":"b5959c78-448e-4322-9535-fbfd4b5efa97","Type":"ContainerStarted","Data":"68642a23cb6f35c664e76681db1e578e0f25522dfb2bfb4a8702d3b8ac3bbdab"}
Sep 16 12:03:26.360727 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.360673    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:58:25 +0000 UTC" deadline="2028-05-01 17:52:15.52434802 +0000 UTC"
Sep 16 12:03:26.360851 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.360729    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14237h48m49.163623281s"
Sep 16 12:03:26.363012 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.362989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dktj" event={"ID":"3ceb08b2-e885-41d1-8e15-7b66decdc24a","Type":"ContainerStarted","Data":"c06de98b6d360598b1a047cd103600079cd9961558ea3d57b8e68067bf669502"}
Sep 16 12:03:26.366604 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.366578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzfbc" event={"ID":"d95554cf-2a64-4e58-8e24-275ebf9ffb78","Type":"ContainerStarted","Data":"33dd9b12dcf4a85264b84d44f451375fb914098f67901f2125dc1b97e5a522b7"}
Sep 16 12:03:26.368800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.368759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"b696e363a3ff242b7722d2f5ae9c04aa44e193c899b7dd300dc5eda88d2cb11a"}
Sep 16 12:03:26.371986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.371514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7lbrw" event={"ID":"ec9781e9-d23e-44bb-a141-1fa6790be59b","Type":"ContainerStarted","Data":"70fde5b432085f8d592ffc79b13edb0b8754f9eff5e7243ded42ec181f1c4c94"}
Sep 16 12:03:26.377198 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.377172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tgsg7" event={"ID":"9a38493c-c3f2-4b9e-8ac4-85bad907ee14","Type":"ContainerStarted","Data":"3872f5c2b91746ed58b9e8e0fd86abcc4bbcd34895100cfb8fe5845dcc5789ba"}
Sep 16 12:03:26.378911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.378874    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" event={"ID":"cc20f8ec2b2204fa94263ebc59d562b0","Type":"ContainerStarted","Data":"9a74e86b51e4b14f19545ff6d57ff6dfce186d6663f3625d820ac2b124cbb4a1"}
Sep 16 12:03:26.380683 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.380653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"cf1a289088fc81807ef16ec01ed9b503","Type":"ContainerStarted","Data":"4bb3cd8ca24752f06e262e51d8a34276e79fb2bbe642e49699863d2ff740e431"}
Sep 16 12:03:26.576672 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.576597    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:26.851936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.851298    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:26.851936 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.851472    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:26.851936 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.851530    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret podName:a4698db6-e801-4511-8142-6a53c3338306 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:28.851513929 +0000 UTC m=+4.986835880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret") pod "global-pull-secret-syncer-pvbqc" (UID: "a4698db6-e801-4511-8142-6a53c3338306") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:26.859507 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.859048    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:26.952509 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:26.952474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:26.952693 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.952665    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:26.952772 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:26.952726    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:28.952707739 +0000 UTC m=+5.088029705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:27.053720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:27.053687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:27.053913 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:27.053894    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:27.053994 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:27.053920    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:27.053994 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:27.053933    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hmsnn for pod openshift-network-diagnostics/network-check-target-q6vwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:27.053994 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:27.053990    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn podName:e4e1f017-a546-4cf7-829d-10ab78aba646 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:29.053972738 +0000 UTC m=+5.189294705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmsnn" (UniqueName: "kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn") pod "network-check-target-q6vwr" (UID: "e4e1f017-a546-4cf7-829d-10ab78aba646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:27.318776 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:27.318700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:27.318957 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:27.318843    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:27.361824 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:27.361771    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:58:25 +0000 UTC" deadline="2028-04-24 15:27:21.346952353 +0000 UTC"
Sep 16 12:03:27.361824 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:27.361823    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14067h23m53.985133914s"
Sep 16 12:03:28.321664 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:28.321638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:28.321850 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:28.321765    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:28.322197 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:28.322178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:28.322291 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:28.322277    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:28.868097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:28.868045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:28.868549 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:28.868184    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:28.868549 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:28.868247    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret podName:a4698db6-e801-4511-8142-6a53c3338306 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:32.868229131 +0000 UTC m=+9.003551086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret") pod "global-pull-secret-syncer-pvbqc" (UID: "a4698db6-e801-4511-8142-6a53c3338306") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:28.968591 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:28.968489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:28.969336 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:28.968916    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:28.969336 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:28.968988    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:32.968968663 +0000 UTC m=+9.104290626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:29.069957 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:29.069859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:29.070113 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:29.069990    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:29.070113 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:29.070011    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:29.070113 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:29.070027    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hmsnn for pod openshift-network-diagnostics/network-check-target-q6vwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:29.070113 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:29.070087    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn podName:e4e1f017-a546-4cf7-829d-10ab78aba646 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:33.070068749 +0000 UTC m=+9.205390704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmsnn" (UniqueName: "kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn") pod "network-check-target-q6vwr" (UID: "e4e1f017-a546-4cf7-829d-10ab78aba646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:29.318842 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:29.318748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:29.319004 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:29.318904    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:30.318865 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:30.318818    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:30.319335 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:30.318976    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:30.319335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:30.319033    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:30.319335 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:30.319183    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:31.319005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:31.318959    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:31.319413 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:31.319114    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:32.319129 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:32.319098    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:32.319561 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:32.319236    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:32.319635 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:32.319606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:32.319711 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:32.319688    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:32.901716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:32.901257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:32.901716 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:32.901399    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:32.901716 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:32.901450    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret podName:a4698db6-e801-4511-8142-6a53c3338306 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:40.901437517 +0000 UTC m=+17.036759470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret") pod "global-pull-secret-syncer-pvbqc" (UID: "a4698db6-e801-4511-8142-6a53c3338306") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:33.002387 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:33.002357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:33.002531 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:33.002516    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:33.002588 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:33.002579    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:41.002559797 +0000 UTC m=+17.137881763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:33.103625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:33.103105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:33.103625 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:33.103289    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:33.103625 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:33.103309    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:33.103625 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:33.103318    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hmsnn for pod openshift-network-diagnostics/network-check-target-q6vwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:33.103625 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:33.103357    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn podName:e4e1f017-a546-4cf7-829d-10ab78aba646 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:41.10334464 +0000 UTC m=+17.238666589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmsnn" (UniqueName: "kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn") pod "network-check-target-q6vwr" (UID: "e4e1f017-a546-4cf7-829d-10ab78aba646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:33.318325 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:33.318233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:33.318480 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:33.318378    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:34.319729 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:34.319703    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:34.320184 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:34.319703    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:34.320184 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:34.319812    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:34.320184 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:34.319914    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:35.318617 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:35.318530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:35.318781 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:35.318663    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:36.321518 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:36.321481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:36.322002 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:36.321601    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:36.322002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:36.321676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:36.322002 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:36.321801    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:37.318565 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:37.318485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:37.318728 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:37.318604    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:38.318619 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:38.318586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:38.319068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:38.318590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:38.319068 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:38.318710    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:38.319068 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:38.318773    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:39.318124 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:39.318094    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:39.318285 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:39.318238    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:40.318753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:40.318719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:40.319190 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:40.318863    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:40.319249 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:40.319207    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:40.319340 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:40.319300    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:40.959934 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:40.959902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:40.960159 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:40.960032    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:40.960159 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:40.960087    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret podName:a4698db6-e801-4511-8142-6a53c3338306 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:56.960070374 +0000 UTC m=+33.095392328 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret") pod "global-pull-secret-syncer-pvbqc" (UID: "a4698db6-e801-4511-8142-6a53c3338306") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:41.064157 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:41.061200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:41.064157 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:41.061312    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:41.064157 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:41.061400    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:57.061376659 +0000 UTC m=+33.196698627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:41.162163 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:41.162123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:41.162333 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:41.162313    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:41.162412 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:41.162343    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:41.162412 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:41.162356    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hmsnn for pod openshift-network-diagnostics/network-check-target-q6vwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:41.162511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:41.162418    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn podName:e4e1f017-a546-4cf7-829d-10ab78aba646 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:57.162398737 +0000 UTC m=+33.297720703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmsnn" (UniqueName: "kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn") pod "network-check-target-q6vwr" (UID: "e4e1f017-a546-4cf7-829d-10ab78aba646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:41.318536 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:41.318445    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:41.318684 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:41.318556    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:42.318683 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:42.318650    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:42.319092 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:42.318773    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:42.319092 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:42.318824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:42.319092 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:42.318920    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:43.318511 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:43.318476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:43.318660 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:43.318601    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:44.320094 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:44.320068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:44.320361 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:44.320162    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:44.320361 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:44.320185    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:44.320361 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:44.320237    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:45.318745 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.318407    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:45.318845 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:45.318747    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:45.423380 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.423351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" event={"ID":"cc20f8ec2b2204fa94263ebc59d562b0","Type":"ContainerStarted","Data":"9a62ef605412a1abcf76517b8c565e802bddc90a9df88f00d70318fd662d3097"}
Sep 16 12:03:45.425043 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.425020    2560 generic.go:356] "Generic (PLEG): container finished" podID="cf1a289088fc81807ef16ec01ed9b503" containerID="e0112d9b18e27059afe866c0dca36b91d996ab8e05403cce8715b9804c3b83bb" exitCode=0
Sep 16 12:03:45.425143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.425102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"cf1a289088fc81807ef16ec01ed9b503","Type":"ContainerDied","Data":"e0112d9b18e27059afe866c0dca36b91d996ab8e05403cce8715b9804c3b83bb"}
Sep 16 12:03:45.426765 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.426736    2560 generic.go:356] "Generic (PLEG): container finished" podID="874c0c70-1ccb-43b6-b102-fa109d487cc5" containerID="c6961b452a99e8feb8a355c55635b4729773359e61df827dca6e2cfc3739701f" exitCode=0
Sep 16 12:03:45.426869 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.426820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerDied","Data":"c6961b452a99e8feb8a355c55635b4729773359e61df827dca6e2cfc3739701f"}
Sep 16 12:03:45.428433 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.428401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbjcp" event={"ID":"205eab7e-100c-46e7-b39b-e3aef2be4adf","Type":"ContainerStarted","Data":"9d347fa3719280e637cd30492ad7568950711923e3587fb725e097decad0efa9"}
Sep 16 12:03:45.429897 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.429884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr" event={"ID":"b5959c78-448e-4322-9535-fbfd4b5efa97","Type":"ContainerStarted","Data":"05ee4eeb9c0f28722883236a37081caf3828841c3a7bfbf5973db263b607716c"}
Sep 16 12:03:45.431332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.431310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dktj" event={"ID":"3ceb08b2-e885-41d1-8e15-7b66decdc24a","Type":"ContainerStarted","Data":"913ca1854d905f963612d475d03ceafb67801c621d0c52710e498f2a508d4d6c"}
Sep 16 12:03:45.432653 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.432629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzfbc" event={"ID":"d95554cf-2a64-4e58-8e24-275ebf9ffb78","Type":"ContainerStarted","Data":"859f6caf71f6041c4c2f06139e7400eb62ea191495998bdba54433ea084bb19b"}
Sep 16 12:03:45.435358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.435340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"4c5cb4a6c8cebb5dbf018083f213ab2dd3ad9a690d9b069102f87770542079e5"}
Sep 16 12:03:45.436285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.435771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"ffbc504e3b1ce20bd59739d9f7c791bb9bf6a3ce8332d2798bbfe55d809e6cee"}
Sep 16 12:03:45.436285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.435873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"59d99b441d637ec3f6d573261736659b33c770f8c784d8b4cefeee45eaf76c68"}
Sep 16 12:03:45.436285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.435966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"92ea3d13424511d358af6ca782bf0c7c577cc1bfec6e99dc719febb879603f9d"}
Sep 16 12:03:45.436285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.436128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"27f33d35f65b6c192ec2bb6cc7fb50a3359da4363d8bca3d96a98b1269b5f737"}
Sep 16 12:03:45.439740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.438868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" podStartSLOduration=20.438850192 podStartE2EDuration="20.438850192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.437360821 +0000 UTC m=+21.572682785" watchObservedRunningTime="2026-09-16 12:03:45.438850192 +0000 UTC m=+21.574172163"
Sep 16 12:03:45.441426 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.441360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7lbrw" event={"ID":"ec9781e9-d23e-44bb-a141-1fa6790be59b","Type":"ContainerStarted","Data":"6ac563c52db8dbdf6bc18099b14dce4081b32c4b90b436bb8bdf6895df109f78"}
Sep 16 12:03:45.443198 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.443177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tgsg7" event={"ID":"9a38493c-c3f2-4b9e-8ac4-85bad907ee14","Type":"ContainerStarted","Data":"5ed674c90b72742aa974c5aed216ce7dfdf7b18f39ca66bbd2348c6361f0ec9e"}
Sep 16 12:03:45.469378 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.469308    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tbjcp" podStartSLOduration=2.633435874 podStartE2EDuration="21.469291336s" totalImagesPullingTime="18.835855462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.450421884 +0000 UTC m=+21.585743855" watchObservedRunningTime="2026-09-16 12:03:45.469291336 +0000 UTC m=+21.604613313"
Sep 16 12:03:45.484951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.484912    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lzfbc" podStartSLOduration=2.730268446 podStartE2EDuration="21.484897746s" totalImagesPullingTime="18.7546293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.484299015 +0000 UTC m=+21.619620990" watchObservedRunningTime="2026-09-16 12:03:45.484897746 +0000 UTC m=+21.620219718"
Sep 16 12:03:45.514987 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.514952    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5dktj" podStartSLOduration=2.694437007 podStartE2EDuration="21.51494112s" totalImagesPullingTime="18.820504113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.514616852 +0000 UTC m=+21.649938822" watchObservedRunningTime="2026-09-16 12:03:45.51494112 +0000 UTC m=+21.650263090"
Sep 16 12:03:45.529463 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.529431    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7lbrw" podStartSLOduration=2.658481611 podStartE2EDuration="21.529419989s" totalImagesPullingTime="18.870938378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.5293248 +0000 UTC m=+21.664646771" watchObservedRunningTime="2026-09-16 12:03:45.529419989 +0000 UTC m=+21.664741959"
Sep 16 12:03:45.545458 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:45.545421    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tgsg7" podStartSLOduration=2.664890356 podStartE2EDuration="21.545409215s" totalImagesPullingTime="18.880518859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:45.545092517 +0000 UTC m=+21.680414488" watchObservedRunningTime="2026-09-16 12:03:45.545409215 +0000 UTC m=+21.680731186"
Sep 16 12:03:46.318580 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:46.318500    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:46.318580 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:46.318533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:46.318875 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:46.318607    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:46.318875 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:46.318729    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:46.447158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:46.447124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"cf1a289088fc81807ef16ec01ed9b503","Type":"ContainerStarted","Data":"97fa34fc29d68e659714654de63182bba280b40c539e2dca83437b52e5127df4"}
Sep 16 12:03:46.449254 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:46.449028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vsmqm" event={"ID":"92db4e9b-0f8f-4dd1-a3f8-bbe9d2ef59a0","Type":"ContainerStarted","Data":"c61f42d0beeb0952714876797b55f7574868dd56300860d0505b5c02305d6dc5"}
Sep 16 12:03:46.452682 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:46.452655    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"9f46341bc60fae372d4d72b29a15d32bc9b75128057d54b9ca5aedc292138965"}
Sep 16 12:03:46.463369 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:46.463318    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" podStartSLOduration=21.463303073 podStartE2EDuration="21.463303073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:46.462355907 +0000 UTC m=+22.597677890" watchObservedRunningTime="2026-09-16 12:03:46.463303073 +0000 UTC m=+22.598625043"
Sep 16 12:03:46.844178 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:46.844157    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:03:47.296126 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:47.296043    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:03:46.844175162Z","UUID":"c25d426d-c2b4-4d09-9d7e-422cf139cfd9","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:03:47.297422 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:47.297406    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:03:47.297510 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:47.297434    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:03:47.318375 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:47.318351    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:47.318481 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:47.318458    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:47.457518 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:47.457486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr" event={"ID":"b5959c78-448e-4322-9535-fbfd4b5efa97","Type":"ContainerStarted","Data":"c0a531d773574a29d9150d745e58c84ecf32d1b98075d70e266a95999eb4b33e"}
Sep 16 12:03:48.318693 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:48.318652    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:48.318882 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:48.318796    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:48.318882 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:48.318823    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:48.319006 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:48.318925    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:48.464178 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:48.464140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"d1f2dde189e10829e9168f0685a9ce534f8fc4ed110929e39755156df757fab8"}
Sep 16 12:03:49.318311 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:49.318285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:49.318490 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:49.318398    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:49.431888 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:49.431851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:49.432744 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:49.432723    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:49.448628 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:49.448569    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vsmqm" podStartSLOduration=6.649400304 podStartE2EDuration="25.448551159s" totalImagesPullingTime="18.799150855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:46.476557764 +0000 UTC m=+22.611879736" watchObservedRunningTime="2026-09-16 12:03:49.448551159 +0000 UTC m=+25.583873130"
Sep 16 12:03:49.466022 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:49.465998    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:49.466409 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:49.466358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lzfbc"
Sep 16 12:03:50.318885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.318612    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:50.318999 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.318667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:50.318999 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:50.318978    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:50.319072 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:50.319030    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:50.469091 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.469068    2560 generic.go:356] "Generic (PLEG): container finished" podID="874c0c70-1ccb-43b6-b102-fa109d487cc5" containerID="f5c489ae4dbe4676fd65bae7cddedf558fa6daac606e626f23262b1a92a5d267" exitCode=0
Sep 16 12:03:50.469588 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.469138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerDied","Data":"f5c489ae4dbe4676fd65bae7cddedf558fa6daac606e626f23262b1a92a5d267"}
Sep 16 12:03:50.471509 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.471488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr" event={"ID":"b5959c78-448e-4322-9535-fbfd4b5efa97","Type":"ContainerStarted","Data":"4f39478c081eddac11070a051f5cc7786afb6b074c9b42499ed9ef9d2df66e33"}
Sep 16 12:03:50.475346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.475319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" event={"ID":"a50f42ac-142a-4dbc-93d0-b8a9a56a9ed6","Type":"ContainerStarted","Data":"c2cde92f7b4d58dfa2a13122a52d07425fd2f05b1efbf51e7b25ef0ffdef0831"}
Sep 16 12:03:50.475590 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.475573    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:50.475730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.475637    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:50.491153 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.491136    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:50.506352 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.506319    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkmgr" podStartSLOduration=3.394270096 podStartE2EDuration="26.506306892s" totalImagesPullingTime="23.112036796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:50.505856097 +0000 UTC m=+26.641178068" watchObservedRunningTime="2026-09-16 12:03:50.506306892 +0000 UTC m=+26.641628862"
Sep 16 12:03:50.532125 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:50.529895    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2" podStartSLOduration=7.315160695 podStartE2EDuration="26.529878558s" totalImagesPullingTime="19.214717863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:50.528755198 +0000 UTC m=+26.664077169" watchObservedRunningTime="2026-09-16 12:03:50.529878558 +0000 UTC m=+26.665200523"
Sep 16 12:03:51.319072 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.319044    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:51.319328 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:51.319185    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:51.478331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.478262    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:51.496163 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.496136    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:03:51.650241 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.650047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdmtp"]
Sep 16 12:03:51.650358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.650339    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:51.650456 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:51.650438    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:51.650971 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.650949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvbqc"]
Sep 16 12:03:51.651049 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.651039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:51.651125 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:51.651111    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:51.652880 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.652847    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q6vwr"]
Sep 16 12:03:51.652967 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:51.652933    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:51.653050 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:51.653026    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:52.484794 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:52.484750    2560 generic.go:356] "Generic (PLEG): container finished" podID="874c0c70-1ccb-43b6-b102-fa109d487cc5" containerID="bd57c5298c08f3304e732c888906435ed228bea158a229b5d90178cbc0dd183d" exitCode=0
Sep 16 12:03:52.485192 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:52.484818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerDied","Data":"bd57c5298c08f3304e732c888906435ed228bea158a229b5d90178cbc0dd183d"}
Sep 16 12:03:53.318512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:53.318450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:53.318512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:53.318483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:53.318512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:53.318483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:53.318874 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:53.318556    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:53.318874 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:53.318690    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:53.318874 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:53.318762    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:53.490198 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:53.490172    2560 generic.go:356] "Generic (PLEG): container finished" podID="874c0c70-1ccb-43b6-b102-fa109d487cc5" containerID="cc8210626927d07c32c165d3cfd21433ab9faa8e0b93bff2208fb2acecd1fa93" exitCode=0
Sep 16 12:03:53.490498 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:53.490206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerDied","Data":"cc8210626927d07c32c165d3cfd21433ab9faa8e0b93bff2208fb2acecd1fa93"}
Sep 16 12:03:55.318609 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:55.318571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:55.318609 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:55.318600    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:55.319152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:55.318701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:55.319152 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:55.318703    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:55.319152 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:55.318818    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:55.319152 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:55.318896    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:56.975001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:56.974965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:56.975427 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:56.975084    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:56.975427 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:56.975142    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret podName:a4698db6-e801-4511-8142-6a53c3338306 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:28.975123783 +0000 UTC m=+65.110445734 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret") pod "global-pull-secret-syncer-pvbqc" (UID: "a4698db6-e801-4511-8142-6a53c3338306") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:57.075660 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.075631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:57.075831 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.075801    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:57.075894 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.075856    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.075837082 +0000 UTC m=+65.211159035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:57.176110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.176077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:57.176259 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.176199    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:57.176259 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.176213    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:57.176259 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.176221    2560 projected.go:196] Error preparing data for projected volume kube-api-access-hmsnn for pod openshift-network-diagnostics/network-check-target-q6vwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:57.176398 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.176266    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn podName:e4e1f017-a546-4cf7-829d-10ab78aba646 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.17625445 +0000 UTC m=+65.311576398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmsnn" (UniqueName: "kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn") pod "network-check-target-q6vwr" (UID: "e4e1f017-a546-4cf7-829d-10ab78aba646") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:57.318556 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.318527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:57.318716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.318575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:57.318716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.318576    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:57.318716 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.318662    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:03:57.318933 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.318779    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvbqc" podUID="a4698db6-e801-4511-8142-6a53c3338306"
Sep 16 12:03:57.318933 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.318872    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q6vwr" podUID="e4e1f017-a546-4cf7-829d-10ab78aba646"
Sep 16 12:03:57.657410 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.657326    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeReady"
Sep 16 12:03:57.657566 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.657472    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:03:57.693337 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.693308    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85b65544b8-8vmkw"]
Sep 16 12:03:57.697236 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.697217    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.699978 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.699877    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:03:57.701454 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.700412    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:03:57.701454 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.700571    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z95hj\""
Sep 16 12:03:57.701454 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.700940    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:03:57.706669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.706648    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"]
Sep 16 12:03:57.707885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.707658    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:03:57.709025 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.709007    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:57.709824 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.709804    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b65544b8-8vmkw"]
Sep 16 12:03:57.711511 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.711486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t4tt2\""
Sep 16 12:03:57.711949 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.711870    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 12:03:57.712906 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.712831    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 12:03:57.713373 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.713245    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-74nwl"]
Sep 16 12:03:57.715535 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.715517    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k8cgc"]
Sep 16 12:03:57.715669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.715648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.717979 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.717960    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:57.718155 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.718137    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:03:57.718155 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.718149    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7d4qr\""
Sep 16 12:03:57.718452 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.718335    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:03:57.720514 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.720499    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nn9m4\""
Sep 16 12:03:57.720833 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.720811    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:03:57.720928 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.720856    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:03:57.720928 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.720883    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:03:57.724720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.724702    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"]
Sep 16 12:03:57.725269 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.725241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8cgc"]
Sep 16 12:03:57.728132 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.728113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-74nwl"]
Sep 16 12:03:57.781387 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-image-registry-private-configuration\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.781526 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-certificates\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.781526 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-trusted-ca\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.781526 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-installation-pull-secrets\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.781685 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsrd\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-kube-api-access-6wsrd\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.781685 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgqdb\" (UniqueName: \"kubernetes.io/projected/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-kube-api-access-qgqdb\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.781685 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-tmp-dir\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.781685 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrwq\" (UniqueName: \"kubernetes.io/projected/7520f5cb-0f5e-417c-8c1a-e3311b7c23ee-kube-api-access-rgrwq\") pod \"ingress-canary-k8cgc\" (UID: \"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee\") " pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:57.781685 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-bound-sa-token\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.781958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7520f5cb-0f5e-417c-8c1a-e3311b7c23ee-cert\") pod \"ingress-canary-k8cgc\" (UID: \"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee\") " pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:57.781958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.781958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-config-volume\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.781958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.781958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:57.781958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:57.781958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.781947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e3c5739-a7c9-4e80-bab5-63637a1c4039-ca-trust-extracted\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.882574 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.882574 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e3c5739-a7c9-4e80-bab5-63637a1c4039-ca-trust-extracted\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-image-registry-private-configuration\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-certificates\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-trusted-ca\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-installation-pull-secrets\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsrd\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-kube-api-access-6wsrd\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.882832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqdb\" (UniqueName: \"kubernetes.io/projected/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-kube-api-access-qgqdb\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.883219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-tmp-dir\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.883219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrwq\" (UniqueName: \"kubernetes.io/projected/7520f5cb-0f5e-417c-8c1a-e3311b7c23ee-kube-api-access-rgrwq\") pod \"ingress-canary-k8cgc\" (UID: \"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee\") " pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:57.883219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-bound-sa-token\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.883219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7520f5cb-0f5e-417c-8c1a-e3311b7c23ee-cert\") pod \"ingress-canary-k8cgc\" (UID: \"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee\") " pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:57.883219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.882978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.883219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.883012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-config-volume\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.883219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.883089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e3c5739-a7c9-4e80-bab5-63637a1c4039-ca-trust-extracted\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.883554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.883358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:57.883554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.883451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-certificates\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.883554 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.882709    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:03:57.883693 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.883574    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:03:58.383556637 +0000 UTC m=+34.518878586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:03:57.883693 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.883573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-config-volume\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.884078 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.884054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-tmp-dir\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.884189 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.882750    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:03:57.884189 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.884162    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:03:57.884189 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.884175    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:03:57.884321 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.884175    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:58.384155017 +0000 UTC m=+34.519476989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:03:57.884321 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:57.884232    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:58.384216298 +0000 UTC m=+34.519538254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:03:57.884558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.884536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-trusted-ca\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.887810 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.887765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-image-registry-private-configuration\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.887963 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.887943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-installation-pull-secrets\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.888065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.887960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7520f5cb-0f5e-417c-8c1a-e3311b7c23ee-cert\") pod \"ingress-canary-k8cgc\" (UID: \"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee\") " pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:57.899998 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.899813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqdb\" (UniqueName: \"kubernetes.io/projected/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-kube-api-access-qgqdb\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:57.899998 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.899955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-bound-sa-token\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:57.900603 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.900130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrwq\" (UniqueName: \"kubernetes.io/projected/7520f5cb-0f5e-417c-8c1a-e3311b7c23ee-kube-api-access-rgrwq\") pod \"ingress-canary-k8cgc\" (UID: \"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee\") " pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:57.900729 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:57.900708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsrd\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-kube-api-access-6wsrd\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:58.036068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.035996    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8cgc"
Sep 16 12:03:58.314933 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.314684    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8cgc"]
Sep 16 12:03:58.318656 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:03:58.318621    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7520f5cb_0f5e_417c_8c1a_e3311b7c23ee.slice/crio-0855623979edcaf787034c6c6a25af1fdbf77e22e067d92d9fa5df6286be8d8e WatchSource:0}: Error finding container 0855623979edcaf787034c6c6a25af1fdbf77e22e067d92d9fa5df6286be8d8e: Status 404 returned error can't find the container with id 0855623979edcaf787034c6c6a25af1fdbf77e22e067d92d9fa5df6286be8d8e
Sep 16 12:03:58.386455 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.386430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:58.386563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.386492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:58.386563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.386534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:58.386671 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:58.386582    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:03:58.386671 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:58.386599    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:03:58.386671 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:58.386618    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:03:58.386671 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:58.386642    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:03:58.386854 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:58.386644    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:59.386629452 +0000 UTC m=+35.521951420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:03:58.386854 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:58.386714    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:03:59.386696422 +0000 UTC m=+35.522018378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:03:58.386854 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:58.386729    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:59.386719714 +0000 UTC m=+35.522041669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:03:58.504268 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.504238    2560 generic.go:356] "Generic (PLEG): container finished" podID="874c0c70-1ccb-43b6-b102-fa109d487cc5" containerID="984e4765f75639c6a7ae6fb26144819614383012343f2fdbb7daa8844bb857e6" exitCode=0
Sep 16 12:03:58.504604 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.504306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerDied","Data":"984e4765f75639c6a7ae6fb26144819614383012343f2fdbb7daa8844bb857e6"}
Sep 16 12:03:58.505686 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:58.505667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8cgc" event={"ID":"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee","Type":"ContainerStarted","Data":"0855623979edcaf787034c6c6a25af1fdbf77e22e067d92d9fa5df6286be8d8e"}
Sep 16 12:03:59.318335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.318305    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:03:59.319089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.318304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:03:59.319089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.318304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:03:59.321153 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.321127    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:03:59.321333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.321214    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:03:59.321333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.321291    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:03:59.322201 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.322183    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:03:59.322289 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.322242    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvq92\""
Sep 16 12:03:59.322404 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.322389    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kdnz\""
Sep 16 12:03:59.394689 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.394663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:03:59.394813 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.394721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:03:59.394813 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.394764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:03:59.394927 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:59.394825    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:03:59.394927 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:59.394842    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:03:59.394927 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:59.394890    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:03:59.394927 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:59.394893    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:03:59.395082 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:59.394901    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:01.394880539 +0000 UTC m=+37.530202491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:03:59.395082 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:59.394968    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:04:01.394952695 +0000 UTC m=+37.530274647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:03:59.395082 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:03:59.394985    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:01.394975249 +0000 UTC m=+37.530297199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:03:59.511729 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.511703    2560 generic.go:356] "Generic (PLEG): container finished" podID="874c0c70-1ccb-43b6-b102-fa109d487cc5" containerID="321e591e151135f4f30f568d703e4217dced1baeb31a2243cdefc1f1cf8ad117" exitCode=0
Sep 16 12:03:59.511849 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:03:59.511733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerDied","Data":"321e591e151135f4f30f568d703e4217dced1baeb31a2243cdefc1f1cf8ad117"}
Sep 16 12:04:00.518258 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:00.518227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45fxl" event={"ID":"874c0c70-1ccb-43b6-b102-fa109d487cc5","Type":"ContainerStarted","Data":"09b907d2f767131494ab5a1ab247f8f67ce91f491ddf607c9de133058365377b"}
Sep 16 12:04:00.546959 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:00.546898    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-45fxl" podStartSLOduration=6.883451261 podStartE2EDuration="36.546878663s" totalImagesPullingTime="29.663427402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:00.54221241 +0000 UTC m=+36.677534384" watchObservedRunningTime="2026-09-16 12:04:00.546878663 +0000 UTC m=+36.682200634"
Sep 16 12:04:01.410918 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:01.410726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:04:01.411080 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:01.410953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:04:01.411080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:01.410887    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:04:01.411080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:01.411051    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:01.411080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:01.411056    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:05.411035384 +0000 UTC m=+41.546357336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:04:01.411080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:01.411062    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:01.411080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:01.411078    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:04:01.411365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:01.410990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:04:01.411365 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:01.411086    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:04:05.411076026 +0000 UTC m=+41.546397991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:01.411365 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:01.411134    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:05.411112732 +0000 UTC m=+41.546434696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:04:01.521896 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:01.521823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8cgc" event={"ID":"7520f5cb-0f5e-417c-8c1a-e3311b7c23ee","Type":"ContainerStarted","Data":"6ab07969a45fb065b65f46632a38c1d4ebb216d750f19eedcce6e445391662e5"}
Sep 16 12:04:01.541149 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:01.541108    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k8cgc" podStartSLOduration=1.936587715 podStartE2EDuration="4.541091318s" totalImagesPullingTime="2.604503603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:01.540863054 +0000 UTC m=+37.676185025" watchObservedRunningTime="2026-09-16 12:04:01.541091318 +0000 UTC m=+37.676413290"
Sep 16 12:04:05.443609 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:05.443572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:05.443629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:05.443655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:05.443736    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:05.443758    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:05.443764    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:05.443779    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:05.443831    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:13.443817116 +0000 UTC m=+49.579139078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:05.443850    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:04:13.443836171 +0000 UTC m=+49.579158145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:05.444065 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:05.443864    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:13.443858584 +0000 UTC m=+49.579180532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:04:13.499751 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:13.499714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:13.499763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:13.499804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:13.499879    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:13.499899    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:13.499910    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:13.499929    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:13.499969    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.499952689 +0000 UTC m=+65.635274640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:13.499982    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.499976644 +0000 UTC m=+65.635298593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:04:13.500168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:13.500005    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:04:29.499990641 +0000 UTC m=+65.635312590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:23.507799 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:23.507757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jw5g2"
Sep 16 12:04:29.014779 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.014726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:04:29.017673 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.017655    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:04:29.028852 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.028826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4698db6-e801-4511-8142-6a53c3338306-original-pull-secret\") pod \"global-pull-secret-syncer-pvbqc\" (UID: \"a4698db6-e801-4511-8142-6a53c3338306\") " pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:04:29.035861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.035840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvbqc"
Sep 16 12:04:29.115432 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.115397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:04:29.117952 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.117936    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:04:29.125989 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.125971    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:04:29.126033 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.126024    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:33.126009381 +0000 UTC m=+129.261331329 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : secret "metrics-daemon-secret" not found
Sep 16 12:04:29.216146 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.216118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:04:29.218583 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.218569    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:04:29.228911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.228892    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:04:29.239295 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.239274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsnn\" (UniqueName: \"kubernetes.io/projected/e4e1f017-a546-4cf7-829d-10ab78aba646-kube-api-access-hmsnn\") pod \"network-check-target-q6vwr\" (UID: \"e4e1f017-a546-4cf7-829d-10ab78aba646\") " pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:04:29.305233 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.305153    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvbqc"]
Sep 16 12:04:29.309046 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:04:29.309014    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4698db6_e801_4511_8142_6a53c3338306.slice/crio-868ed7c0b08547a17357f1dae88b287fb2e99b54b3e2d8ace1921227d89823df WatchSource:0}: Error finding container 868ed7c0b08547a17357f1dae88b287fb2e99b54b3e2d8ace1921227d89823df: Status 404 returned error can't find the container with id 868ed7c0b08547a17357f1dae88b287fb2e99b54b3e2d8ace1921227d89823df
Sep 16 12:04:29.332166 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.332146    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvq92\""
Sep 16 12:04:29.340195 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.340176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:04:29.518021 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.517986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:04:29.518176 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.518035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:04:29.518176 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.518060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:04:29.518176 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.518157    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:04:29.518176 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.518163    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:04:29.518176 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.518179    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:04:29.518330 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.518189    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:29.518330 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.518212    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:01.518197482 +0000 UTC m=+97.653519444 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:04:29.518330 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.518224    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:01.518219142 +0000 UTC m=+97.653541091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:04:29.518330 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:04:29.518235    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:05:01.518229278 +0000 UTC m=+97.653551227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:29.587034 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.586964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvbqc" event={"ID":"a4698db6-e801-4511-8142-6a53c3338306","Type":"ContainerStarted","Data":"868ed7c0b08547a17357f1dae88b287fb2e99b54b3e2d8ace1921227d89823df"}
Sep 16 12:04:29.607349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:29.607310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q6vwr"]
Sep 16 12:04:29.611031 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:04:29.611004    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e1f017_a546_4cf7_829d_10ab78aba646.slice/crio-6cf72e23398c654162363e6925cc0828701b41393d7149f1456ce1cc1136e92e WatchSource:0}: Error finding container 6cf72e23398c654162363e6925cc0828701b41393d7149f1456ce1cc1136e92e: Status 404 returned error can't find the container with id 6cf72e23398c654162363e6925cc0828701b41393d7149f1456ce1cc1136e92e
Sep 16 12:04:30.592423 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:30.592383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q6vwr" event={"ID":"e4e1f017-a546-4cf7-829d-10ab78aba646","Type":"ContainerStarted","Data":"6cf72e23398c654162363e6925cc0828701b41393d7149f1456ce1cc1136e92e"}
Sep 16 12:04:34.605208 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:34.605015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvbqc" event={"ID":"a4698db6-e801-4511-8142-6a53c3338306","Type":"ContainerStarted","Data":"c7eb9145ce7cb5eb774b43f1a3911374374711282514b1aa1c1f4ee0c0243f9e"}
Sep 16 12:04:34.606652 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:34.606623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q6vwr" event={"ID":"e4e1f017-a546-4cf7-829d-10ab78aba646","Type":"ContainerStarted","Data":"4f12cb87d23b96e0414fd2068286c788f30b95d4c9163ae20468a73631f6f909"}
Sep 16 12:04:34.606731 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:34.606684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:04:34.620514 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:34.620475    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pvbqc" podStartSLOduration=65.506452446 podStartE2EDuration="1m10.620462925s" totalImagesPullingTime="5.114010479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:34.619384017 +0000 UTC m=+70.754705988" watchObservedRunningTime="2026-09-16 12:04:34.620462925 +0000 UTC m=+70.755784895"
Sep 16 12:04:34.650831 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:04:34.650771    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q6vwr" podStartSLOduration=65.840340171 podStartE2EDuration="1m10.650753382s" totalImagesPullingTime="4.810413211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:34.649883367 +0000 UTC m=+70.785205338" watchObservedRunningTime="2026-09-16 12:04:34.650753382 +0000 UTC m=+70.786075355"
Sep 16 12:05:01.534018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:05:01.533919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:05:01.534018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:05:01.533982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:05:01.534018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:05:01.534004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:05:01.534511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:01.534079    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:05:01.534511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:01.534100    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:05:01.534511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:01.534100    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:05:01.534511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:01.534116    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:05:01.534511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:01.534155    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:06:05.534142597 +0000 UTC m=+161.669464545 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:05:01.534511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:01.534168    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:06:05.534162882 +0000 UTC m=+161.669484830 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:05:01.534511 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:01.534178    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:06:05.534172424 +0000 UTC m=+161.669494373 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:05:05.612068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:05:05.612033    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q6vwr"
Sep 16 12:05:33.151396 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:05:33.151362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:05:33.152012 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:33.151517    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:05:33.152012 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:05:33.151602    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:35.151581916 +0000 UTC m=+251.286903868 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : secret "metrics-daemon-secret" not found
Sep 16 12:06:00.711461 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:00.711419    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039"
Sep 16 12:06:00.722530 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:00.722508    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh" podUID="c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90"
Sep 16 12:06:00.729715 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:00.729685    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-74nwl" podUID="97a89fdd-e02c-454c-8a5d-787e2b5a5c8f"
Sep 16 12:06:00.812562 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:06:00.812528    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:06:00.812562 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:06:00.812541    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74nwl"
Sep 16 12:06:00.812761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:06:00.812542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:06:02.342475 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:02.342439    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:06:05.567997 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:06:05.567953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:06:05.567997 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:06:05.568015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:05.568101    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:05.568108    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:05.568130    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85b65544b8-8vmkw: secret "image-registry-tls" not found
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:05.568154    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls podName:97a89fdd-e02c-454c-8a5d-787e2b5a5c8f nodeName:}" failed. No retries permitted until 2026-09-16 12:08:07.568139991 +0000 UTC m=+283.703461940 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls") pod "dns-default-74nwl" (UID: "97a89fdd-e02c-454c-8a5d-787e2b5a5c8f") : secret "dns-default-metrics-tls" not found
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:06:05.568158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:05.568185    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls podName:4e3c5739-a7c9-4e80-bab5-63637a1c4039 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:07.56817083 +0000 UTC m=+283.703492784 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls") pod "image-registry-85b65544b8-8vmkw" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039") : secret "image-registry-tls" not found
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:05.568244    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:06:05.568591 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:06:05.568310    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:07.568294931 +0000 UTC m=+283.703616893 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:06:13.318639 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:06:13.318603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:07:35.165520 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:35.165478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:07:35.166117 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:35.165641    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:07:35.166117 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:35.165726    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs podName:3360bde0-4498-4c93-85ca-0813afa0fc95 nodeName:}" failed. No retries permitted until 2026-09-16 12:09:37.165707196 +0000 UTC m=+373.301029194 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs") pod "network-metrics-daemon-sdmtp" (UID: "3360bde0-4498-4c93-85ca-0813afa0fc95") : secret "metrics-daemon-secret" not found
Sep 16 12:07:52.811674 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.811584    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"]
Sep 16 12:07:52.813507 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.813484    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:52.814708 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.814677    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-rkmf9"]
Sep 16 12:07:52.816057 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.816025    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hq6qb\""
Sep 16 12:07:52.816456 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.816438    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5bb97b79b-6jf4w"]
Sep 16 12:07:52.816572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.816555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:52.817463 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.817436    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 12:07:52.817575 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.817462    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.817575 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.817479    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 12:07:52.817575 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.817462    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.818158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.818138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:52.819135 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.819114    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 12:07:52.819625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.819606    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.819704 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.819627    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.820765 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.820743    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jlvxh\""
Sep 16 12:07:52.821100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.821048    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.821340 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.821318    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8vxs9\""
Sep 16 12:07:52.822036 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.821728    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 12:07:52.822036 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.821746    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 12:07:52.822036 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.822032    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.822257 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.822228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 12:07:52.822583 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.822565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 12:07:52.822631 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.822599    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 12:07:52.826366 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.826326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"]
Sep 16 12:07:52.832165 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.832134    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-rkmf9"]
Sep 16 12:07:52.832494 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.832472    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 12:07:52.836873 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.836850    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb97b79b-6jf4w"]
Sep 16 12:07:52.881231 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.881202    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"]
Sep 16 12:07:52.882751 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.882737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"
Sep 16 12:07:52.885187 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.885159    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7lbjk\""
Sep 16 12:07:52.886995 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.886974    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-kdfjs"]
Sep 16 12:07:52.888895 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.888702    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:52.894081 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.894058    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-82bb6\""
Sep 16 12:07:52.894927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.894906    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"]
Sep 16 12:07:52.895379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.895322    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.896360 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.896342    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.896468 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.896437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 12:07:52.896677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.896660    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 12:07:52.901022 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.901003    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 12:07:52.905407 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.905384    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kdfjs"]
Sep 16 12:07:52.983661 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-stats-auth\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:52.983827 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzl4k\" (UniqueName: \"kubernetes.io/projected/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-kube-api-access-pzl4k\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:52.983827 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t6c\" (UniqueName: \"kubernetes.io/projected/bec56780-8565-43f0-9159-57e83ca48bed-kube-api-access-b6t6c\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:52.983827 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7pzn\" (UniqueName: \"kubernetes.io/projected/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-kube-api-access-h7pzn\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:52.983827 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:52.984040 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-tmp\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:52.984040 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:52.984040 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.983952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:52.984040 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-config\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:52.984040 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-service-ca-bundle\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlb5\" (UniqueName: \"kubernetes.io/projected/94577f34-a9a4-4f64-bfeb-08c30a90fc25-kube-api-access-6mlb5\") pod \"network-check-source-6f8b768455-c8bmg\" (UID: \"94577f34-a9a4-4f64-bfeb-08c30a90fc25\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-default-certificate\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-serving-cert\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-snapshots\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chrv\" (UniqueName: \"kubernetes.io/projected/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-kube-api-access-2chrv\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:52.984298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-trusted-ca\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:52.984574 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:52.984336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-serving-cert\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.085114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-config\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.085114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-service-ca-bundle\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.085114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlb5\" (UniqueName: \"kubernetes.io/projected/94577f34-a9a4-4f64-bfeb-08c30a90fc25-kube-api-access-6mlb5\") pod \"network-check-source-6f8b768455-c8bmg\" (UID: \"94577f34-a9a4-4f64-bfeb-08c30a90fc25\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"
Sep 16 12:07:53.085114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-default-certificate\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.085334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.085334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-serving-cert\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.085334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-snapshots\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.085334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chrv\" (UniqueName: \"kubernetes.io/projected/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-kube-api-access-2chrv\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.085334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:53.085334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-trusted-ca\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.085334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-serving-cert\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-stats-auth\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzl4k\" (UniqueName: \"kubernetes.io/projected/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-kube-api-access-pzl4k\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t6c\" (UniqueName: \"kubernetes.io/projected/bec56780-8565-43f0-9159-57e83ca48bed-kube-api-access-b6t6c\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pzn\" (UniqueName: \"kubernetes.io/projected/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-kube-api-access-h7pzn\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-tmp\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:53.085670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.086080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.085717    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:07:53.086080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.085774    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:07:53.585756572 +0000 UTC m=+269.721078535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : secret "router-metrics-certs-default" not found
Sep 16 12:07:53.086080 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.085970    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:07:53.585953542 +0000 UTC m=+269.721275510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : configmap references non-existent config key: service-ca.crt
Sep 16 12:07:53.086080 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.085996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-service-ca-bundle\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.086080 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.086054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-config\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.086389 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.086155    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:53.086389 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.086177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:53.086389 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.086221    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls podName:8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc nodeName:}" failed. No retries permitted until 2026-09-16 12:07:53.586203053 +0000 UTC m=+269.721525002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-22g9k" (UID: "8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:53.086389 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.086270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-tmp\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.086389 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.086278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-trusted-ca\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.087056 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.087024    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.087247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.087223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-snapshots\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.088100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.088073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-default-certificate\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.088492 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.088473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-serving-cert\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.088952 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.088928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-serving-cert\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.089034 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.089020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-stats-auth\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.094284 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.094239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pzn\" (UniqueName: \"kubernetes.io/projected/a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24-kube-api-access-h7pzn\") pod \"insights-operator-5844b856db-kdfjs\" (UID: \"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24\") " pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.094927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.094904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t6c\" (UniqueName: \"kubernetes.io/projected/bec56780-8565-43f0-9159-57e83ca48bed-kube-api-access-b6t6c\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.095064 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.094982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chrv\" (UniqueName: \"kubernetes.io/projected/0bc44c33-1937-4555-bbe5-de6ea2a19c8c-kube-api-access-2chrv\") pod \"console-operator-696884fb9c-rkmf9\" (UID: \"0bc44c33-1937-4555-bbe5-de6ea2a19c8c\") " pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.095134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.095067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlb5\" (UniqueName: \"kubernetes.io/projected/94577f34-a9a4-4f64-bfeb-08c30a90fc25-kube-api-access-6mlb5\") pod \"network-check-source-6f8b768455-c8bmg\" (UID: \"94577f34-a9a4-4f64-bfeb-08c30a90fc25\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"
Sep 16 12:07:53.096112 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.096089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzl4k\" (UniqueName: \"kubernetes.io/projected/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-kube-api-access-pzl4k\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:53.139508 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.139487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:07:53.194146 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.194118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"
Sep 16 12:07:53.201043 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.201019    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kdfjs"
Sep 16 12:07:53.406670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.406641    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-rkmf9"]
Sep 16 12:07:53.408948 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:07:53.408927    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc44c33_1937_4555_bbe5_de6ea2a19c8c.slice/crio-77053685c5630ad40608fa45a6abce42b4e341e0a2f8f73fb62f9cd3b9d24aaf WatchSource:0}: Error finding container 77053685c5630ad40608fa45a6abce42b4e341e0a2f8f73fb62f9cd3b9d24aaf: Status 404 returned error can't find the container with id 77053685c5630ad40608fa45a6abce42b4e341e0a2f8f73fb62f9cd3b9d24aaf
Sep 16 12:07:53.462979 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.462958    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg"]
Sep 16 12:07:53.465459 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:07:53.465438    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94577f34_a9a4_4f64_bfeb_08c30a90fc25.slice/crio-43c388ea2ad8cc2c3b66d367b370056856a27ca58e7a58881626e331c2793b06 WatchSource:0}: Error finding container 43c388ea2ad8cc2c3b66d367b370056856a27ca58e7a58881626e331c2793b06: Status 404 returned error can't find the container with id 43c388ea2ad8cc2c3b66d367b370056856a27ca58e7a58881626e331c2793b06
Sep 16 12:07:53.469624 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.469599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kdfjs"]
Sep 16 12:07:53.474126 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:07:53.474105    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81fa552_7a8e_4ae4_9c7a_ad8f11ae1a24.slice/crio-eb25466a304f0b2e5796d7bc33f30b8369715639e3076ae3308f8b4dc950a62f WatchSource:0}: Error finding container eb25466a304f0b2e5796d7bc33f30b8369715639e3076ae3308f8b4dc950a62f: Status 404 returned error can't find the container with id eb25466a304f0b2e5796d7bc33f30b8369715639e3076ae3308f8b4dc950a62f
Sep 16 12:07:53.589889 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.589858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.590008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.589946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:53.590076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:53.590009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:53.590076 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.590062    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:53.590172 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.590079    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:07:54.590058835 +0000 UTC m=+270.725380787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : configmap references non-existent config key: service-ca.crt
Sep 16 12:07:53.590172 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.590133    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls podName:8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc nodeName:}" failed. No retries permitted until 2026-09-16 12:07:54.590114536 +0000 UTC m=+270.725436488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-22g9k" (UID: "8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:53.590172 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.590140    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:07:53.590278 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:53.590188    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:07:54.590175019 +0000 UTC m=+270.725496984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : secret "router-metrics-certs-default" not found
Sep 16 12:07:54.083322 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.083289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kdfjs" event={"ID":"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24","Type":"ContainerStarted","Data":"eb25466a304f0b2e5796d7bc33f30b8369715639e3076ae3308f8b4dc950a62f"}
Sep 16 12:07:54.084595 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.084560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" event={"ID":"0bc44c33-1937-4555-bbe5-de6ea2a19c8c","Type":"ContainerStarted","Data":"77053685c5630ad40608fa45a6abce42b4e341e0a2f8f73fb62f9cd3b9d24aaf"}
Sep 16 12:07:54.086162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.086135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg" event={"ID":"94577f34-a9a4-4f64-bfeb-08c30a90fc25","Type":"ContainerStarted","Data":"3aa4d8cba08b3c24c0dd6ad8e37946c06cfb47e8898a1ee5335461a2ba72b3f4"}
Sep 16 12:07:54.086267 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.086180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg" event={"ID":"94577f34-a9a4-4f64-bfeb-08c30a90fc25","Type":"ContainerStarted","Data":"43c388ea2ad8cc2c3b66d367b370056856a27ca58e7a58881626e331c2793b06"}
Sep 16 12:07:54.102258 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.102218    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8bmg" podStartSLOduration=2.102204104 podStartE2EDuration="2.102204104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:54.101548158 +0000 UTC m=+270.236870131" watchObservedRunningTime="2026-09-16 12:07:54.102204104 +0000 UTC m=+270.237526079"
Sep 16 12:07:54.598129 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.598094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:54.598314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.598146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:54.598314 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:54.598216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:54.598314 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:54.598309    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:07:56.598288446 +0000 UTC m=+272.733610474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : configmap references non-existent config key: service-ca.crt
Sep 16 12:07:54.598479 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:54.598374    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:07:54.598479 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:54.598422    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:07:56.598401824 +0000 UTC m=+272.733723774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : secret "router-metrics-certs-default" not found
Sep 16 12:07:54.598479 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:54.598469    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:54.598625 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:54.598537    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls podName:8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc nodeName:}" failed. No retries permitted until 2026-09-16 12:07:56.598519948 +0000 UTC m=+272.733841913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-22g9k" (UID: "8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:56.093599 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.093574    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/0.log"
Sep 16 12:07:56.094018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.093615    2560 generic.go:356] "Generic (PLEG): container finished" podID="0bc44c33-1937-4555-bbe5-de6ea2a19c8c" containerID="ca998acb360c004bb9186ec7d4d6f81fd1e68ec86aa0c52ee11416f96170d55f" exitCode=255
Sep 16 12:07:56.094018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.093716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" event={"ID":"0bc44c33-1937-4555-bbe5-de6ea2a19c8c","Type":"ContainerDied","Data":"ca998acb360c004bb9186ec7d4d6f81fd1e68ec86aa0c52ee11416f96170d55f"}
Sep 16 12:07:56.094018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.093941    2560 scope.go:117] "RemoveContainer" containerID="ca998acb360c004bb9186ec7d4d6f81fd1e68ec86aa0c52ee11416f96170d55f"
Sep 16 12:07:56.095426 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.095394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kdfjs" event={"ID":"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24","Type":"ContainerStarted","Data":"a4be6a17899114ad759fc884fc92bdc1b8a97400d7710782e3c74d7ef0bab930"}
Sep 16 12:07:56.126670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.126621    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-kdfjs" podStartSLOduration=1.765044238 podStartE2EDuration="4.126601763s" totalImagesPullingTime="2.361557525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:56.124478585 +0000 UTC m=+272.259800556" watchObservedRunningTime="2026-09-16 12:07:56.126601763 +0000 UTC m=+272.261923736"
Sep 16 12:07:56.614132 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.614103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:56.614253 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.614150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:07:56.614253 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.614210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:07:56.614331 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:56.614265    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.614246876 +0000 UTC m=+276.749568845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : configmap references non-existent config key: service-ca.crt
Sep 16 12:07:56.614331 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:56.614300    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:56.614411 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:56.614331    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:07:56.614411 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:56.614365    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls podName:8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.614352759 +0000 UTC m=+276.749674713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-22g9k" (UID: "8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:07:56.614411 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:56.614379    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.614372716 +0000 UTC m=+276.749694665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : secret "router-metrics-certs-default" not found
Sep 16 12:07:56.855271 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.855238    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7bd29"]
Sep 16 12:07:56.857241 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.857223    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:56.859929 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.859894    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:07:56.859929 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.859899    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:07:56.860232 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.860215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hqfk\""
Sep 16 12:07:56.867426 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:56.867382    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bd29"]
Sep 16 12:07:57.018096 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.018074    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnrn\" (UniqueName: \"kubernetes.io/projected/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-api-access-6lnrn\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.018211 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.018113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.018211 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.018163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.018211 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.018195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90740c99-e643-46f8-8d06-b9c67ab4b81b-crio-socket\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.018369 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.018300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90740c99-e643-46f8-8d06-b9c67ab4b81b-data-volume\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.100377 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.100358    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:07:57.100823 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.100808    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/0.log"
Sep 16 12:07:57.100879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.100845    2560 generic.go:356] "Generic (PLEG): container finished" podID="0bc44c33-1937-4555-bbe5-de6ea2a19c8c" containerID="0241ec86b5f88d5f0ad98fd965201e1a7b208a9aea8f95fe36ea7c93bb8abcac" exitCode=255
Sep 16 12:07:57.100914 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.100876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" event={"ID":"0bc44c33-1937-4555-bbe5-de6ea2a19c8c","Type":"ContainerDied","Data":"0241ec86b5f88d5f0ad98fd965201e1a7b208a9aea8f95fe36ea7c93bb8abcac"}
Sep 16 12:07:57.100943 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.100929    2560 scope.go:117] "RemoveContainer" containerID="ca998acb360c004bb9186ec7d4d6f81fd1e68ec86aa0c52ee11416f96170d55f"
Sep 16 12:07:57.101137 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.101114    2560 scope.go:117] "RemoveContainer" containerID="0241ec86b5f88d5f0ad98fd965201e1a7b208a9aea8f95fe36ea7c93bb8abcac"
Sep 16 12:07:57.101336 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.101315    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-rkmf9_openshift-console-operator(0bc44c33-1937-4555-bbe5-de6ea2a19c8c)\"" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" podUID="0bc44c33-1937-4555-bbe5-de6ea2a19c8c"
Sep 16 12:07:57.118759 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.118712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90740c99-e643-46f8-8d06-b9c67ab4b81b-data-volume\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.118759 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.118750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnrn\" (UniqueName: \"kubernetes.io/projected/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-api-access-6lnrn\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.118892 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.118775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.118892 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.118840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.118892 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.118874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90740c99-e643-46f8-8d06-b9c67ab4b81b-crio-socket\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.119011 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.118927    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.119011 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.118966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90740c99-e643-46f8-8d06-b9c67ab4b81b-crio-socket\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.119011 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.118981    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:07:57.618962844 +0000 UTC m=+273.754284809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.119011 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.118983    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.119165 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.119029    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:07:57.61901672 +0000 UTC m=+273.754338694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.119165 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.119065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90740c99-e643-46f8-8d06-b9c67ab4b81b-data-volume\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.132224 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.132204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnrn\" (UniqueName: \"kubernetes.io/projected/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-api-access-6lnrn\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.623334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.623304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.623478 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:57.623361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:57.623478 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.623412    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.623585 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.623480    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:07:58.623461934 +0000 UTC m=+274.758783897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.623585 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.623486    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.623585 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:57.623533    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:07:58.623516163 +0000 UTC m=+274.758838111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:58.110067 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:58.110039    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:07:58.110419 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:58.110358    2560 scope.go:117] "RemoveContainer" containerID="0241ec86b5f88d5f0ad98fd965201e1a7b208a9aea8f95fe36ea7c93bb8abcac"
Sep 16 12:07:58.110526 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:58.110505    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-rkmf9_openshift-console-operator(0bc44c33-1937-4555-bbe5-de6ea2a19c8c)\"" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" podUID="0bc44c33-1937-4555-bbe5-de6ea2a19c8c"
Sep 16 12:07:58.626965 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:58.626935    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7lbrw_ec9781e9-d23e-44bb-a141-1fa6790be59b/dns-node-resolver/0.log"
Sep 16 12:07:58.631225 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:58.631198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:58.631290 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:58.631267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:07:58.631349 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:58.631309    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:58.631397 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:58.631372    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.631353838 +0000 UTC m=+276.766675808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:58.631461 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:58.631429    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:58.631521 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:07:58.631508    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.631487793 +0000 UTC m=+276.766809800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:59.827405 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:07:59.827374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbjcp_205eab7e-100c-46e7-b39b-e3aef2be4adf/node-ca/0.log"
Sep 16 12:08:00.430234 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.430203    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9x7j6"]
Sep 16 12:08:00.432831 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.432812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.435575 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.435554    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 12:08:00.435703 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.435649    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 12:08:00.435864 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.435848    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 12:08:00.436927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.436908    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6kxlz\""
Sep 16 12:08:00.437034 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.436932    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 12:08:00.441721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.441690    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9x7j6"]
Sep 16 12:08:00.545192 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.545171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxz4\" (UniqueName: \"kubernetes.io/projected/eef04105-0f41-430a-aabd-56c7d9085a08-kube-api-access-kkxz4\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.545285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.545211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eef04105-0f41-430a-aabd-56c7d9085a08-signing-key\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.545285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.545240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eef04105-0f41-430a-aabd-56c7d9085a08-signing-cabundle\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.627428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.627402    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8cgc_7520f5cb-0f5e-417c-8c1a-e3311b7c23ee/serve-healthcheck-canary/0.log"
Sep 16 12:08:00.645754 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.645731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:00.645844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.645770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:00.645844 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.645820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:08:00.645969 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.645888    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:08:00.645969 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.645915    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:08:08.645892244 +0000 UTC m=+284.781214213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : configmap references non-existent config key: service-ca.crt
Sep 16 12:08:00.645969 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.645938    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:08:00.645969 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.645953    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:08:04.645935266 +0000 UTC m=+280.781257238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : configmap "kube-rbac-proxy" not found
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.645985    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls podName:8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc nodeName:}" failed. No retries permitted until 2026-09-16 12:08:08.645971307 +0000 UTC m=+284.781293298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-22g9k" (UID: "8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.645980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.646025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.646058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxz4\" (UniqueName: \"kubernetes.io/projected/eef04105-0f41-430a-aabd-56c7d9085a08-kube-api-access-kkxz4\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.646056    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.646105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eef04105-0f41-430a-aabd-56c7d9085a08-signing-key\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.646138    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:08:00.646168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.646152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eef04105-0f41-430a-aabd-56c7d9085a08-signing-cabundle\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.646500 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.646186    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:08:04.646164319 +0000 UTC m=+280.781486268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:00.646500 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:00.646228    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:08:08.64620339 +0000 UTC m=+284.781525342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : secret "router-metrics-certs-default" not found
Sep 16 12:08:00.646897 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.646878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eef04105-0f41-430a-aabd-56c7d9085a08-signing-cabundle\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.648750 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.648731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eef04105-0f41-430a-aabd-56c7d9085a08-signing-key\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.654612 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.654587    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxz4\" (UniqueName: \"kubernetes.io/projected/eef04105-0f41-430a-aabd-56c7d9085a08-kube-api-access-kkxz4\") pod \"service-ca-568869fdc5-9x7j6\" (UID: \"eef04105-0f41-430a-aabd-56c7d9085a08\") " pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:00.743368 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:00.743311    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9x7j6"
Sep 16 12:08:01.015958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:01.015884    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9x7j6"]
Sep 16 12:08:01.019156 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:01.019128    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef04105_0f41_430a_aabd_56c7d9085a08.slice/crio-e0d80b6588c7a533fbbbd2d850f5fb1686ca22fd8091e4c99ca9e714a221343a WatchSource:0}: Error finding container e0d80b6588c7a533fbbbd2d850f5fb1686ca22fd8091e4c99ca9e714a221343a: Status 404 returned error can't find the container with id e0d80b6588c7a533fbbbd2d850f5fb1686ca22fd8091e4c99ca9e714a221343a
Sep 16 12:08:01.119817 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:01.119757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9x7j6" event={"ID":"eef04105-0f41-430a-aabd-56c7d9085a08","Type":"ContainerStarted","Data":"e0d80b6588c7a533fbbbd2d850f5fb1686ca22fd8091e4c99ca9e714a221343a"}
Sep 16 12:08:03.126916 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:03.126882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9x7j6" event={"ID":"eef04105-0f41-430a-aabd-56c7d9085a08","Type":"ContainerStarted","Data":"687184d7279f9c1e6b7e171e12a495b70a719b93e17772a2746a06d2ebf630be"}
Sep 16 12:08:03.139691 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:03.139674    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:08:03.139756 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:03.139713    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:08:03.144004 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:03.140347    2560 scope.go:117] "RemoveContainer" containerID="0241ec86b5f88d5f0ad98fd965201e1a7b208a9aea8f95fe36ea7c93bb8abcac"
Sep 16 12:08:03.144004 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:03.140597    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-rkmf9_openshift-console-operator(0bc44c33-1937-4555-bbe5-de6ea2a19c8c)\"" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" podUID="0bc44c33-1937-4555-bbe5-de6ea2a19c8c"
Sep 16 12:08:03.147390 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:03.147352    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-9x7j6" podStartSLOduration=1.43306529 podStartE2EDuration="3.147335348s" totalImagesPullingTime="1.714270058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:03.146694858 +0000 UTC m=+279.282016830" watchObservedRunningTime="2026-09-16 12:08:03.147335348 +0000 UTC m=+279.282657359"
Sep 16 12:08:03.813574 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:03.813530    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh" podUID="c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90"
Sep 16 12:08:03.813823 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:03.813529    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-74nwl" podUID="97a89fdd-e02c-454c-8a5d-787e2b5a5c8f"
Sep 16 12:08:03.813926 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:03.813657    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039"
Sep 16 12:08:04.130730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:04.130707    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74nwl"
Sep 16 12:08:04.131297 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:04.130707    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:08:04.131297 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:04.130882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:08:04.679456 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:04.679425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:04.679609 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:04.679483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:04.679609 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:04.679566    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:08:04.679671 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:04.679636    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:04.679671 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:04.679655    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:08:12.679634238 +0000 UTC m=+288.814956202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : configmap "kube-rbac-proxy" not found
Sep 16 12:08:04.679770 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:04.679677    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:08:12.679665676 +0000 UTC m=+288.814987625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:07.603299 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.603268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:08:07.603841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.603315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:08:07.603841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.603370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:08:07.603841 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:07.603482    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:08:07.603841 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:07.603565    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert podName:c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90 nodeName:}" failed. No retries permitted until 2026-09-16 12:10:09.603543466 +0000 UTC m=+405.738865431 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j52wh" (UID: "c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90") : secret "networking-console-plugin-cert" not found
Sep 16 12:08:07.605747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.605724    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97a89fdd-e02c-454c-8a5d-787e2b5a5c8f-metrics-tls\") pod \"dns-default-74nwl\" (UID: \"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f\") " pod="openshift-dns/dns-default-74nwl"
Sep 16 12:08:07.605994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.605974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"image-registry-85b65544b8-8vmkw\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") " pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:08:07.734594 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.734569    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7d4qr\""
Sep 16 12:08:07.735746 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.735726    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z95hj\""
Sep 16 12:08:07.742851 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.742832    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:08:07.743005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:07.742987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74nwl"
Sep 16 12:08:08.021955 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.021890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-74nwl"]
Sep 16 12:08:08.025071 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.025047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85b65544b8-8vmkw"]
Sep 16 12:08:08.025276 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:08.025251    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a89fdd_e02c_454c_8a5d_787e2b5a5c8f.slice/crio-c4d7df06e1ef4165dd84a0cac5e074aa222a9da95bae665c6e6ad93b90aa1fa8 WatchSource:0}: Error finding container c4d7df06e1ef4165dd84a0cac5e074aa222a9da95bae665c6e6ad93b90aa1fa8: Status 404 returned error can't find the container with id c4d7df06e1ef4165dd84a0cac5e074aa222a9da95bae665c6e6ad93b90aa1fa8
Sep 16 12:08:08.144648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.144612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-74nwl" event={"ID":"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f","Type":"ContainerStarted","Data":"c4d7df06e1ef4165dd84a0cac5e074aa222a9da95bae665c6e6ad93b90aa1fa8"}
Sep 16 12:08:08.146391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.146368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" event={"ID":"4e3c5739-a7c9-4e80-bab5-63637a1c4039","Type":"ContainerStarted","Data":"20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108"}
Sep 16 12:08:08.146516 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.146404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" event={"ID":"4e3c5739-a7c9-4e80-bab5-63637a1c4039","Type":"ContainerStarted","Data":"550b427c89ff4830b7037e40d193eaad6cfcc2295c223be1a484129ce51ecb64"}
Sep 16 12:08:08.146516 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.146506    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:08:08.170718 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.170671    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" podStartSLOduration=284.170657395 podStartE2EDuration="4m44.170657395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:08.169575069 +0000 UTC m=+284.304897040" watchObservedRunningTime="2026-09-16 12:08:08.170657395 +0000 UTC m=+284.305979366"
Sep 16 12:08:08.712644 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.712610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:08:08.713123 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.712694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:08.713123 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:08.712777    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:08:08.713123 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:08.712810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:08.713123 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:08.712861    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls podName:8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc nodeName:}" failed. No retries permitted until 2026-09-16 12:08:24.712843328 +0000 UTC m=+300.848165277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-22g9k" (UID: "8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:08:08.713123 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:08.712868    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:08:08.713123 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:08.712929    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:08:24.712911919 +0000 UTC m=+300.848233874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : secret "router-metrics-certs-default" not found
Sep 16 12:08:08.713123 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:08.712947    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle podName:bec56780-8565-43f0-9159-57e83ca48bed nodeName:}" failed. No retries permitted until 2026-09-16 12:08:24.712938239 +0000 UTC m=+300.848260193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle") pod "router-default-5bb97b79b-6jf4w" (UID: "bec56780-8565-43f0-9159-57e83ca48bed") : configmap references non-existent config key: service-ca.crt
Sep 16 12:08:10.155272 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:10.155172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-74nwl" event={"ID":"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f","Type":"ContainerStarted","Data":"85e4e9d10e36013f0a525ec5dbc3bbcba2734e8bc38407aa8c6e0d1c9f58393a"}
Sep 16 12:08:10.155272 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:10.155238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-74nwl" event={"ID":"97a89fdd-e02c-454c-8a5d-787e2b5a5c8f","Type":"ContainerStarted","Data":"4a1dc04f779daaaea4341611a7a697bae98ef2c8ca0c0b60b42b6250991aa9a7"}
Sep 16 12:08:10.155850 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:10.155376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-74nwl"
Sep 16 12:08:10.175656 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:10.175579    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-74nwl" podStartSLOduration=251.486326881 podStartE2EDuration="4m13.175562285s" totalImagesPullingTime="1.689235404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:10.17483221 +0000 UTC m=+286.310154182" watchObservedRunningTime="2026-09-16 12:08:10.175562285 +0000 UTC m=+286.310884259"
Sep 16 12:08:12.747960 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:12.747914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:12.748335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:12.747984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:12.748335 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:12.748147    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:12.748335 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:12.748223    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls podName:90740c99-e643-46f8-8d06-b9c67ab4b81b nodeName:}" failed. No retries permitted until 2026-09-16 12:08:28.748203662 +0000 UTC m=+304.883525617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bd29" (UID: "90740c99-e643-46f8-8d06-b9c67ab4b81b") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:12.748461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:12.748376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90740c99-e643-46f8-8d06-b9c67ab4b81b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:16.319916 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:16.319874    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sdmtp" podUID="3360bde0-4498-4c93-85ca-0813afa0fc95"
Sep 16 12:08:18.318914 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:18.318884    2560 scope.go:117] "RemoveContainer" containerID="0241ec86b5f88d5f0ad98fd965201e1a7b208a9aea8f95fe36ea7c93bb8abcac"
Sep 16 12:08:19.184711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:19.184682    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:08:19.184905 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:19.184742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" event={"ID":"0bc44c33-1937-4555-bbe5-de6ea2a19c8c","Type":"ContainerStarted","Data":"e40d50aaaddc4dbc5dfe2b07a2765bfdd84edd3e61d3b3cd41498d0f5616c42e"}
Sep 16 12:08:19.185073 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:19.185051    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:08:19.189639 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:19.189620    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9"
Sep 16 12:08:19.201933 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:19.201890    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-rkmf9" podStartSLOduration=24.775216028 podStartE2EDuration="27.20187848s" totalImagesPullingTime="2.426662452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:19.201210126 +0000 UTC m=+295.336532098" watchObservedRunningTime="2026-09-16 12:08:19.20187848 +0000 UTC m=+295.337200450"
Sep 16 12:08:20.161303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:20.161269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-74nwl"
Sep 16 12:08:24.234474 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.234449    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:08:24.235455 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.235430    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:08:24.248423 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.248398    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:08:24.735144 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.735099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:24.735144 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.735152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:08:24.736577 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.735326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:24.736577 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.735839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bec56780-8565-43f0-9159-57e83ca48bed-service-ca-bundle\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:24.737636 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.737615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-22g9k\" (UID: \"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:08:24.737741 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.737681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec56780-8565-43f0-9159-57e83ca48bed-metrics-certs\") pod \"router-default-5bb97b79b-6jf4w\" (UID: \"bec56780-8565-43f0-9159-57e83ca48bed\") " pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:24.927937 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.927912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hq6qb\""
Sep 16 12:08:24.936163 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.936144    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"
Sep 16 12:08:24.937742 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.937710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8vxs9\""
Sep 16 12:08:24.946192 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:24.946172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:25.218374 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:25.218343    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k"]
Sep 16 12:08:25.221196 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:25.221171    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd8f8b9_4b8b_45f4_bb54_d14ba2cb68bc.slice/crio-0eff19d205799aa8c40734a3953ed4c767cedca9f79bd98b7f5c2f48658c7392 WatchSource:0}: Error finding container 0eff19d205799aa8c40734a3953ed4c767cedca9f79bd98b7f5c2f48658c7392: Status 404 returned error can't find the container with id 0eff19d205799aa8c40734a3953ed4c767cedca9f79bd98b7f5c2f48658c7392
Sep 16 12:08:25.227003 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:25.226985    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb97b79b-6jf4w"]
Sep 16 12:08:25.229126 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:25.229108    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec56780_8565_43f0_9159_57e83ca48bed.slice/crio-a1d016efb8dfc2c968c55dab4b2921c0d70b32b78ee5c05bfa2d7791ab97e808 WatchSource:0}: Error finding container a1d016efb8dfc2c968c55dab4b2921c0d70b32b78ee5c05bfa2d7791ab97e808: Status 404 returned error can't find the container with id a1d016efb8dfc2c968c55dab4b2921c0d70b32b78ee5c05bfa2d7791ab97e808
Sep 16 12:08:26.207631 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:26.207589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k" event={"ID":"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc","Type":"ContainerStarted","Data":"0eff19d205799aa8c40734a3953ed4c767cedca9f79bd98b7f5c2f48658c7392"}
Sep 16 12:08:26.209221 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:26.209195    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb97b79b-6jf4w" event={"ID":"bec56780-8565-43f0-9159-57e83ca48bed","Type":"ContainerStarted","Data":"3de5a869c5ee34fbed481e6f500c215d4fc3454de476348a1a89c8c3aa25b8b1"}
Sep 16 12:08:26.209338 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:26.209236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb97b79b-6jf4w" event={"ID":"bec56780-8565-43f0-9159-57e83ca48bed","Type":"ContainerStarted","Data":"a1d016efb8dfc2c968c55dab4b2921c0d70b32b78ee5c05bfa2d7791ab97e808"}
Sep 16 12:08:26.228051 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:26.228003    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5bb97b79b-6jf4w" podStartSLOduration=34.227983441 podStartE2EDuration="34.227983441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:26.226409688 +0000 UTC m=+302.361731660" watchObservedRunningTime="2026-09-16 12:08:26.227983441 +0000 UTC m=+302.363305415"
Sep 16 12:08:26.947428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:26.947393    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:26.950063 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:26.950042    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:27.214834 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:27.214734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k" event={"ID":"8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc","Type":"ContainerStarted","Data":"6ad94582ec741fb45a6efa1813fb466d1a169dfbed539047b82e8b78ee40f67a"}
Sep 16 12:08:27.215280 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:27.214999    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:27.216205 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:27.216181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5bb97b79b-6jf4w"
Sep 16 12:08:27.238504 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:27.234543    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-22g9k" podStartSLOduration=33.506705 podStartE2EDuration="35.234523362s" totalImagesPullingTime="1.727818362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:27.23311326 +0000 UTC m=+303.368435230" watchObservedRunningTime="2026-09-16 12:08:27.234523362 +0000 UTC m=+303.369845334"
Sep 16 12:08:27.746923 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:27.746888    2560 patch_prober.go:28] interesting pod/image-registry-85b65544b8-8vmkw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:08:27.747104 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:27.746941    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:08:28.767510 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:28.767463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:28.770022 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:28.770002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90740c99-e643-46f8-8d06-b9c67ab4b81b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bd29\" (UID: \"90740c99-e643-46f8-8d06-b9c67ab4b81b\") " pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:28.969095 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:28.969063    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hqfk\""
Sep 16 12:08:28.977255 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:28.977234    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bd29"
Sep 16 12:08:29.154571 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:29.154547    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:08:29.245023 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:29.244995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bd29"]
Sep 16 12:08:29.247834 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:29.247811    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90740c99_e643_46f8_8d06_b9c67ab4b81b.slice/crio-1a38813d4332a9cbcb0f939971a38df0f615b9dc2c5fe3e6ccb5db8f85002f23 WatchSource:0}: Error finding container 1a38813d4332a9cbcb0f939971a38df0f615b9dc2c5fe3e6ccb5db8f85002f23: Status 404 returned error can't find the container with id 1a38813d4332a9cbcb0f939971a38df0f615b9dc2c5fe3e6ccb5db8f85002f23
Sep 16 12:08:29.307209 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:29.307190    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:08:29.319089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:29.319056    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:08:30.226685 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:30.226627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bd29" event={"ID":"90740c99-e643-46f8-8d06-b9c67ab4b81b","Type":"ContainerStarted","Data":"361c6d43c4589231c6389918db3bb0cafbb1d88e2514669416aa003d0cfacdf7"}
Sep 16 12:08:30.226685 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:30.226678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bd29" event={"ID":"90740c99-e643-46f8-8d06-b9c67ab4b81b","Type":"ContainerStarted","Data":"d0d27ddec9f97db940d6d1d3911479b87563c66af7bdd3e8cefe343313ea2847"}
Sep 16 12:08:30.227015 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:30.226696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bd29" event={"ID":"90740c99-e643-46f8-8d06-b9c67ab4b81b","Type":"ContainerStarted","Data":"1a38813d4332a9cbcb0f939971a38df0f615b9dc2c5fe3e6ccb5db8f85002f23"}
Sep 16 12:08:31.918446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.918415    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b87455d7-pqbjl"]
Sep 16 12:08:31.921513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.921498    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:31.925534 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925512    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:08:31.925687 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925529    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:08:31.925836 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925551    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:08:31.925836 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:08:31.925836 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925768    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:08:31.926002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925634    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:08:31.926002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925645    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:08:31.926002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.925594    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-982qg\""
Sep 16 12:08:31.932668 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.932650    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b87455d7-pqbjl"]
Sep 16 12:08:31.992213 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.992184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-console-config\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:31.992303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.992237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-oauth-config\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:31.992303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.992269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-oauth-serving-cert\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:31.992420 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.992300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnp9\" (UniqueName: \"kubernetes.io/projected/dc981f02-9682-4560-a586-606b4fd46cb6-kube-api-access-7mnp9\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:31.992478 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.992431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-service-ca\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:31.992478 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:31.992464    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-serving-cert\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.093189 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.093165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnp9\" (UniqueName: \"kubernetes.io/projected/dc981f02-9682-4560-a586-606b4fd46cb6-kube-api-access-7mnp9\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.093269 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.093227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-service-ca\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.093269 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.093258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-serving-cert\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.093369 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.093284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-console-config\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.093369 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.093322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-oauth-config\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.093369 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.093346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-oauth-serving-cert\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.093885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.093858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-service-ca\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.094053 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.094034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-oauth-serving-cert\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.094096 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.094064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-console-config\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.095720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.095700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-serving-cert\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.095869 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.095856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-oauth-config\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.103108 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.103089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnp9\" (UniqueName: \"kubernetes.io/projected/dc981f02-9682-4560-a586-606b4fd46cb6-kube-api-access-7mnp9\") pod \"console-b87455d7-pqbjl\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") " pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.230197 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.230138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:32.235032 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.235008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bd29" event={"ID":"90740c99-e643-46f8-8d06-b9c67ab4b81b","Type":"ContainerStarted","Data":"39b9168ebb134bd4b332a5475aad4b777d2346e65929420a1a9a821e2539141b"}
Sep 16 12:08:32.285611 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.285553    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7bd29" podStartSLOduration=34.360473609 podStartE2EDuration="36.28553288s" totalImagesPullingTime="1.925059271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:07:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:32.283583232 +0000 UTC m=+308.418905217" watchObservedRunningTime="2026-09-16 12:08:32.28553288 +0000 UTC m=+308.420854855"
Sep 16 12:08:32.466458 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.466430    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"]
Sep 16 12:08:32.470816 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.470776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.473612 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.473584    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 12:08:32.473735 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.473586    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 12:08:32.473735 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.473618    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:08:32.473735 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.473644    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9fcj7\""
Sep 16 12:08:32.480230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.480178    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"]
Sep 16 12:08:32.501088 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.501051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b87455d7-pqbjl"]
Sep 16 12:08:32.503968 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:32.503943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc981f02_9682_4560_a586_606b4fd46cb6.slice/crio-84d49ef94bca8473a54f8c07d5cbce9b20f7b3bba80725e54aea9f11fd0cabb2 WatchSource:0}: Error finding container 84d49ef94bca8473a54f8c07d5cbce9b20f7b3bba80725e54aea9f11fd0cabb2: Status 404 returned error can't find the container with id 84d49ef94bca8473a54f8c07d5cbce9b20f7b3bba80725e54aea9f11fd0cabb2
Sep 16 12:08:32.598120 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.598094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e827b4-8130-4a82-bbd4-3fab17dc154c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.598209 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.598130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.598209 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.598163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.598209 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.598188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxlkt\" (UniqueName: \"kubernetes.io/projected/50e827b4-8130-4a82-bbd4-3fab17dc154c-kube-api-access-zxlkt\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.698749 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.698723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e827b4-8130-4a82-bbd4-3fab17dc154c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.698858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.698760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.698858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.698809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.698858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.698836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxlkt\" (UniqueName: \"kubernetes.io/projected/50e827b4-8130-4a82-bbd4-3fab17dc154c-kube-api-access-zxlkt\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.699023 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:32.698968    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 12:08:32.699079 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:32.699036    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-tls podName:50e827b4-8130-4a82-bbd4-3fab17dc154c nodeName:}" failed. No retries permitted until 2026-09-16 12:08:33.199015595 +0000 UTC m=+309.334337548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-nvqbq" (UID: "50e827b4-8130-4a82-bbd4-3fab17dc154c") : secret "prometheus-operator-tls" not found
Sep 16 12:08:32.699347 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.699326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e827b4-8130-4a82-bbd4-3fab17dc154c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.701129 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.701109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:32.709541 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:32.709521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxlkt\" (UniqueName: \"kubernetes.io/projected/50e827b4-8130-4a82-bbd4-3fab17dc154c-kube-api-access-zxlkt\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:33.202555 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:33.202518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:33.205396 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:33.205359    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50e827b4-8130-4a82-bbd4-3fab17dc154c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-nvqbq\" (UID: \"50e827b4-8130-4a82-bbd4-3fab17dc154c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:33.240841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:33.240810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b87455d7-pqbjl" event={"ID":"dc981f02-9682-4560-a586-606b4fd46cb6","Type":"ContainerStarted","Data":"84d49ef94bca8473a54f8c07d5cbce9b20f7b3bba80725e54aea9f11fd0cabb2"}
Sep 16 12:08:33.380881 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:33.380853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"
Sep 16 12:08:33.655270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:33.655216    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq"]
Sep 16 12:08:33.657765 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:33.657732    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e827b4_8130_4a82_bbd4_3fab17dc154c.slice/crio-dcee4d8489bbce89e3ffc60f7ecce296011d24f1e430d7bc404353009a167e17 WatchSource:0}: Error finding container dcee4d8489bbce89e3ffc60f7ecce296011d24f1e430d7bc404353009a167e17: Status 404 returned error can't find the container with id dcee4d8489bbce89e3ffc60f7ecce296011d24f1e430d7bc404353009a167e17
Sep 16 12:08:34.246377 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:34.246333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq" event={"ID":"50e827b4-8130-4a82-bbd4-3fab17dc154c","Type":"ContainerStarted","Data":"dcee4d8489bbce89e3ffc60f7ecce296011d24f1e430d7bc404353009a167e17"}
Sep 16 12:08:36.254950 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:36.254915    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq" event={"ID":"50e827b4-8130-4a82-bbd4-3fab17dc154c","Type":"ContainerStarted","Data":"9ce58a6c696fc07f50c054b211c0d2d1e7e1240703ba067c2e21be2191c572ba"}
Sep 16 12:08:36.255421 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:36.254976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq" event={"ID":"50e827b4-8130-4a82-bbd4-3fab17dc154c","Type":"ContainerStarted","Data":"3ae6c9e68eaa0ba59287f1c257b8ddcdc3d314cc3e7b54d64f782f91f45f29bb"}
Sep 16 12:08:36.256571 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:36.256545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b87455d7-pqbjl" event={"ID":"dc981f02-9682-4560-a586-606b4fd46cb6","Type":"ContainerStarted","Data":"64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d"}
Sep 16 12:08:36.273308 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:36.273266    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nvqbq" podStartSLOduration=2.328002614 podStartE2EDuration="4.273252342s" totalImagesPullingTime="1.945249728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:36.272319951 +0000 UTC m=+312.407641921" watchObservedRunningTime="2026-09-16 12:08:36.273252342 +0000 UTC m=+312.408574313"
Sep 16 12:08:36.289319 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:36.289274    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b87455d7-pqbjl" podStartSLOduration=2.187568058 podStartE2EDuration="5.289260605s" totalImagesPullingTime="3.101692547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:36.289001527 +0000 UTC m=+312.424323511" watchObservedRunningTime="2026-09-16 12:08:36.289260605 +0000 UTC m=+312.424582576"
Sep 16 12:08:37.855553 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.855518    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"]
Sep 16 12:08:37.859825 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.859778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:37.863927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.863721    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 12:08:37.863927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.863772    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 12:08:37.863927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.863803    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mm4r4\""
Sep 16 12:08:37.864145 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.863862    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 12:08:37.871846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.871824    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"]
Sep 16 12:08:37.879508 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.879471    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nkpnv"]
Sep 16 12:08:37.884064 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.884043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:37.888295 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.888261    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:08:37.888748 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.888550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvlt5\""
Sep 16 12:08:37.888862 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.888777    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:08:37.889707 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.889109    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:08:37.940027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.939996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:37.940169 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.940042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9daf85a9-d24f-4b15-b09b-868e5999d3db-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:37.940169 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.940065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:37.940169 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.940082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26cb\" (UniqueName: \"kubernetes.io/projected/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-api-access-k26cb\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:37.940333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.940214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:37.940333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:37.940241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9daf85a9-d24f-4b15-b09b-868e5999d3db-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.040543 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gm65\" (UniqueName: \"kubernetes.io/projected/b4eae29c-5100-4f8e-ae4a-5cfba555e551-kube-api-access-5gm65\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.040543 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-sys\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.040739 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9daf85a9-d24f-4b15-b09b-868e5999d3db-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.040739 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.040739 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9daf85a9-d24f-4b15-b09b-868e5999d3db-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.040739 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.040953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26cb\" (UniqueName: \"kubernetes.io/projected/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-api-access-k26cb\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.040953 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:38.040773    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 12:08:38.040953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.040836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-textfile\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.040953 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:08:38.040851    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-tls podName:9daf85a9-d24f-4b15-b09b-868e5999d3db nodeName:}" failed. No retries permitted until 2026-09-16 12:08:38.540830473 +0000 UTC m=+314.676152439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-ftbc8" (UID: "9daf85a9-d24f-4b15-b09b-868e5999d3db") : secret "kube-state-metrics-tls" not found
Sep 16 12:08:38.041147 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-tls\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.041200 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-root\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.041200 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9daf85a9-d24f-4b15-b09b-868e5999d3db-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.041288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-wtmp\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.041348 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4eae29c-5100-4f8e-ae4a-5cfba555e551-metrics-client-ca\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.041401 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.041401 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.041492 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.041492 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.041590 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.041396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9daf85a9-d24f-4b15-b09b-868e5999d3db-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.044052 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.044017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.051902 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.051884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26cb\" (UniqueName: \"kubernetes.io/projected/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-api-access-k26cb\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.142375 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.142311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-wtmp\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.142375 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.142370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4eae29c-5100-4f8e-ae4a-5cfba555e551-metrics-client-ca\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.142562 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.142528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-wtmp\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.142719 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.142654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.142913 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.142883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.143106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.142901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4eae29c-5100-4f8e-ae4a-5cfba555e551-metrics-client-ca\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.143197 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.143151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.143295 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.143049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gm65\" (UniqueName: \"kubernetes.io/projected/b4eae29c-5100-4f8e-ae4a-5cfba555e551-kube-api-access-5gm65\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.143447 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.143432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-sys\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.143825 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.143739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-sys\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.144074 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.144040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-textfile\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.144226 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.144211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-tls\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.144669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.144651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-root\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.144938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.144337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-textfile\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.144938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.144729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4eae29c-5100-4f8e-ae4a-5cfba555e551-root\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.145970 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.145929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.147214 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.147174    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4eae29c-5100-4f8e-ae4a-5cfba555e551-node-exporter-tls\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.151530 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.151507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gm65\" (UniqueName: \"kubernetes.io/projected/b4eae29c-5100-4f8e-ae4a-5cfba555e551-kube-api-access-5gm65\") pod \"node-exporter-nkpnv\" (UID: \"b4eae29c-5100-4f8e-ae4a-5cfba555e551\") " pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.200559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.200536    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nkpnv"
Sep 16 12:08:38.212889 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:38.212862    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4eae29c_5100_4f8e_ae4a_5cfba555e551.slice/crio-a3d2e660672282ca61109e6443cc5bd6140b23aac17061410a9d875058bb76c6 WatchSource:0}: Error finding container a3d2e660672282ca61109e6443cc5bd6140b23aac17061410a9d875058bb76c6: Status 404 returned error can't find the container with id a3d2e660672282ca61109e6443cc5bd6140b23aac17061410a9d875058bb76c6
Sep 16 12:08:38.265289 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.265256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkpnv" event={"ID":"b4eae29c-5100-4f8e-ae4a-5cfba555e551","Type":"ContainerStarted","Data":"a3d2e660672282ca61109e6443cc5bd6140b23aac17061410a9d875058bb76c6"}
Sep 16 12:08:38.549710 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.549630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.552158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.552140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9daf85a9-d24f-4b15-b09b-868e5999d3db-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-ftbc8\" (UID: \"9daf85a9-d24f-4b15-b09b-868e5999d3db\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:38.770645 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:38.770621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"
Sep 16 12:08:39.052018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:39.051907    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8"]
Sep 16 12:08:39.058072 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:39.058044    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9daf85a9_d24f_4b15_b09b_868e5999d3db.slice/crio-f569ab57904840d8d3aded119dd7a169a191c30e9363cd603d58b30387b61136 WatchSource:0}: Error finding container f569ab57904840d8d3aded119dd7a169a191c30e9363cd603d58b30387b61136: Status 404 returned error can't find the container with id f569ab57904840d8d3aded119dd7a169a191c30e9363cd603d58b30387b61136
Sep 16 12:08:39.270722 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:39.270686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8" event={"ID":"9daf85a9-d24f-4b15-b09b-868e5999d3db","Type":"ContainerStarted","Data":"f569ab57904840d8d3aded119dd7a169a191c30e9363cd603d58b30387b61136"}
Sep 16 12:08:40.276493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.276459    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4eae29c-5100-4f8e-ae4a-5cfba555e551" containerID="0e02b2f72c9a6529d4a06d1bec476b4c978456a6f2b520b09beda48e3ea3406b" exitCode=0
Sep 16 12:08:40.276937 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.276546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkpnv" event={"ID":"b4eae29c-5100-4f8e-ae4a-5cfba555e551","Type":"ContainerDied","Data":"0e02b2f72c9a6529d4a06d1bec476b4c978456a6f2b520b09beda48e3ea3406b"}
Sep 16 12:08:40.859684 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.859627    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75b474c465-8rpdm"]
Sep 16 12:08:40.863893 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.863871    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.866585 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.866557    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 12:08:40.866716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.866562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 12:08:40.866716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.866610    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 12:08:40.866716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.866563    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-epcj74ueeqb23\""
Sep 16 12:08:40.867032 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.867016    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 12:08:40.867100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.867021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lp92j\""
Sep 16 12:08:40.867100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.867055    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 12:08:40.879524 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.879498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75b474c465-8rpdm"]
Sep 16 12:08:40.967684 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-tls\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.967879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.967879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d16564e-095d-4135-b3a0-78f92c472b35-metrics-client-ca\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.967879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58cz\" (UniqueName: \"kubernetes.io/projected/2d16564e-095d-4135-b3a0-78f92c472b35-kube-api-access-j58cz\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.967879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.967879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.967879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-grpc-tls\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:40.968134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:40.967912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.068811 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.068754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-tls\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.068938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.068825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.068938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.068852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d16564e-095d-4135-b3a0-78f92c472b35-metrics-client-ca\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.068938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.068875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58cz\" (UniqueName: \"kubernetes.io/projected/2d16564e-095d-4135-b3a0-78f92c472b35-kube-api-access-j58cz\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.069087 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.068937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.069087 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.068965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.069194 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.069176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-grpc-tls\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.069249 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.069230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.069663 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.069640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d16564e-095d-4135-b3a0-78f92c472b35-metrics-client-ca\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.072002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.071976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.072217 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.072199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.072264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.072208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.072264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.072208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-tls\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.072335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.072312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.072409 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.072394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d16564e-095d-4135-b3a0-78f92c472b35-secret-grpc-tls\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.076590 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.076573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58cz\" (UniqueName: \"kubernetes.io/projected/2d16564e-095d-4135-b3a0-78f92c472b35-kube-api-access-j58cz\") pod \"thanos-querier-75b474c465-8rpdm\" (UID: \"2d16564e-095d-4135-b3a0-78f92c472b35\") " pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.175047 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.174985    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:41.282014 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.281975    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkpnv" event={"ID":"b4eae29c-5100-4f8e-ae4a-5cfba555e551","Type":"ContainerStarted","Data":"4c9885c2f1fe0fd7ab7cfd5287d5137edc0d397425bba3dd673e2480218e3814"}
Sep 16 12:08:41.282397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.282044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkpnv" event={"ID":"b4eae29c-5100-4f8e-ae4a-5cfba555e551","Type":"ContainerStarted","Data":"e0664a1aad43b2f8c064394a3a965602597a83452dac7043a26537cfbad417b5"}
Sep 16 12:08:41.284050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.284023    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8" event={"ID":"9daf85a9-d24f-4b15-b09b-868e5999d3db","Type":"ContainerStarted","Data":"9f13da471e56dff6b4245f6c2733de4315ce1bab29af9bb324038fec7fa7b73e"}
Sep 16 12:08:41.284140 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.284081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8" event={"ID":"9daf85a9-d24f-4b15-b09b-868e5999d3db","Type":"ContainerStarted","Data":"4f5e4f117a342c31ce49c2c43c676a43f0fd8dff4a9e10423f13e31aa4bd714d"}
Sep 16 12:08:41.284140 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.284105    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8" event={"ID":"9daf85a9-d24f-4b15-b09b-868e5999d3db","Type":"ContainerStarted","Data":"0000af199dfae36a468702be693963b8bbf9a54a6fb7fd0fe7c172378b0983fe"}
Sep 16 12:08:41.300505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.300453    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nkpnv" podStartSLOduration=3.02052702 podStartE2EDuration="4.300434856s" totalImagesPullingTime="1.211875289s" totalInitContainerRuntime="68.032547ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:41.298634025 +0000 UTC m=+317.433955996" watchObservedRunningTime="2026-09-16 12:08:41.300434856 +0000 UTC m=+317.435756828"
Sep 16 12:08:41.318105 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.318066    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ftbc8" podStartSLOduration=3.092553867 podStartE2EDuration="4.318055966s" totalImagesPullingTime="1.225502099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:41.316688971 +0000 UTC m=+317.452010943" watchObservedRunningTime="2026-09-16 12:08:41.318055966 +0000 UTC m=+317.453377936"
Sep 16 12:08:41.450698 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:41.450637    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75b474c465-8rpdm"]
Sep 16 12:08:41.453390 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:41.453364    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d16564e_095d_4135_b3a0_78f92c472b35.slice/crio-e659fb4d3d343361f67ae49a0a63365674996ba9602b709dca451b9363cc625a WatchSource:0}: Error finding container e659fb4d3d343361f67ae49a0a63365674996ba9602b709dca451b9363cc625a: Status 404 returned error can't find the container with id e659fb4d3d343361f67ae49a0a63365674996ba9602b709dca451b9363cc625a
Sep 16 12:08:42.231010 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:42.230975    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:42.231217 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:42.231041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:42.237247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:42.237222    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:42.288958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:42.288922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" event={"ID":"2d16564e-095d-4135-b3a0-78f92c472b35","Type":"ContainerStarted","Data":"e659fb4d3d343361f67ae49a0a63365674996ba9602b709dca451b9363cc625a"}
Sep 16 12:08:42.293378 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:42.293354    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:08:44.079693 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.079659    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:08:44.082874 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.082851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.086935 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.086492    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 12:08:44.086935 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.086716    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 12:08:44.087126 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087108    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 12:08:44.087182 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087157    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 12:08:44.087235 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087185    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dw2jk\""
Sep 16 12:08:44.087400 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087379    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 12:08:44.087581 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087566    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 12:08:44.087673 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087642    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 12:08:44.087673 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087665    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 12:08:44.087763 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.087670    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 12:08:44.090571 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.090549    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:08:44.090719 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.090699    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 12:08:44.090832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.090760    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 12:08:44.096477 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.094468    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1jbh1uj4b2m4f\""
Sep 16 12:08:44.098559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.098537    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 12:08:44.100465 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.100443    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 12:08:44.101920 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.101903    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 12:08:44.105925 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.105908    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:08:44.201156 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201156 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72sh7\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-kube-api-access-72sh7\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-config\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201429    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201678 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201678 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201678 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201678 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201678 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.201851 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.201680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.299854 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.299818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" event={"ID":"2d16564e-095d-4135-b3a0-78f92c472b35","Type":"ContainerStarted","Data":"2460343ad914b7c65d75fafa93e7f884931b111b9d776cd6b77888a0db62b3b6"}
Sep 16 12:08:44.299986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.299890    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" event={"ID":"2d16564e-095d-4135-b3a0-78f92c472b35","Type":"ContainerStarted","Data":"6ad9dc1fa8aa5a83751b6a193bcdc265e14a9a73deb29a488d355304ee812155"}
Sep 16 12:08:44.299986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.299926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" event={"ID":"2d16564e-095d-4135-b3a0-78f92c472b35","Type":"ContainerStarted","Data":"4a06d58c4231429263bba69744bd19b1cea7b9af5779eefd83bf7fda5c0462f4"}
Sep 16 12:08:44.302302 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.302274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.302858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.302831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.302951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.302874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-config\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.302951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.302916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.302951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.302938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.302966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.302995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303304 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303304 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303304 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303304 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303304 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303304 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303399    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.303579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.303451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72sh7\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-kube-api-access-72sh7\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.304164 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.304132    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.304643 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.304580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.304643 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.304619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.304815 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.304701    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.305682 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.305351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.305682 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.305543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.306458 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.305985    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.308353 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.308323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.308353 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.308343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.308523 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.308423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-config\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.308853 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.308829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.309387 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.309364    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.309485 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.309384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.309976 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.309954    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.310207 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.310188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.310303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.310280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.310739 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.310670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.311959 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.311813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.311959 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.311915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.315955 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.315932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72sh7\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-kube-api-access-72sh7\") pod \"prometheus-k8s-0\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.395858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.395731    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:08:44.686346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:44.686323    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:08:44.690646 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:08:44.690616    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb645edd_ae24_436a_8dc6_f16ec7e05441.slice/crio-96abde73ca939d1171a8452a5669685e31518fdfeb44bea56bf474055608440d WatchSource:0}: Error finding container 96abde73ca939d1171a8452a5669685e31518fdfeb44bea56bf474055608440d: Status 404 returned error can't find the container with id 96abde73ca939d1171a8452a5669685e31518fdfeb44bea56bf474055608440d
Sep 16 12:08:45.308279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:45.308245    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" event={"ID":"2d16564e-095d-4135-b3a0-78f92c472b35","Type":"ContainerStarted","Data":"8af0285ce036af355fde515d4e2c73dc377070a19ff9d8aaa0f210dcd50209d4"}
Sep 16 12:08:45.308724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:45.308330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" event={"ID":"2d16564e-095d-4135-b3a0-78f92c472b35","Type":"ContainerStarted","Data":"2ca5b4d573cc26cd914cf7425a3502ed8d6c34504d142811b0bee9a326b8bcd8"}
Sep 16 12:08:45.308724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:45.308388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" event={"ID":"2d16564e-095d-4135-b3a0-78f92c472b35","Type":"ContainerStarted","Data":"573c8e5c3449736cb265f7cca101c8de6e9d6e2577dad92a5ab748545f37d82d"}
Sep 16 12:08:45.308724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:45.308433    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:45.310393 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:45.310363    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerStarted","Data":"96abde73ca939d1171a8452a5669685e31518fdfeb44bea56bf474055608440d"}
Sep 16 12:08:45.338526 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:45.338469    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm" podStartSLOduration=2.291947794 podStartE2EDuration="5.338449902s" totalImagesPullingTime="3.046502108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:45.33497151 +0000 UTC m=+321.470293484" watchObservedRunningTime="2026-09-16 12:08:45.338449902 +0000 UTC m=+321.473771874"
Sep 16 12:08:45.817213 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:45.817190    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b65544b8-8vmkw"]
Sep 16 12:08:46.315966 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:46.315934    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerID="d528a3cb05e8bbe31ce6e34347c71417a8b8debed20de5891b31e1db83f8c4bc" exitCode=0
Sep 16 12:08:46.316399 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:46.316027    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"d528a3cb05e8bbe31ce6e34347c71417a8b8debed20de5891b31e1db83f8c4bc"}
Sep 16 12:08:50.338266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:50.338202    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerStarted","Data":"ae20ccc023a062c5edb5a2e19d254883cee6af10a74e0112ed0266e7177a7202"}
Sep 16 12:08:50.338266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:50.338263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerStarted","Data":"3ad882dea5a29ad5b0b721f814262dcba3c040a0508c7a4d5810990cb08fff1a"}
Sep 16 12:08:50.338686 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:50.338288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerStarted","Data":"984bfef77f43a104c2f3088e4b3427f23e576b97841c98e953bfe5322b5aecc1"}
Sep 16 12:08:50.338686 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:50.338317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerStarted","Data":"080b68d656a79c7af68646f6db338f2405219bac3d6543d83b1d47d8e5075e66"}
Sep 16 12:08:50.338686 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:50.338343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerStarted","Data":"c813f9e5f3e2a27b418ed99381161bc80d048738fa92dcb4815ee5f6f4afa2dc"}
Sep 16 12:08:50.338686 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:50.338367    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerStarted","Data":"f196bc8211acb2aa0e0b4a0f7011e995076cbaf510dc84179fb30dd744914766"}
Sep 16 12:08:50.365576 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:50.365524    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.8416539410000001 podStartE2EDuration="6.365509975s" totalImagesPullingTime="4.504491101s" totalInitContainerRuntime="19.364933ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:50.363919228 +0000 UTC m=+326.499241222" watchObservedRunningTime="2026-09-16 12:08:50.365509975 +0000 UTC m=+326.500832016"
Sep 16 12:08:51.322065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:51.322040    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75b474c465-8rpdm"
Sep 16 12:08:54.396744 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:08:54.396714    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:09:02.095141 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:02.095107    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b87455d7-pqbjl"]
Sep 16 12:09:10.836004 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:10.835955    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039" containerName="registry" containerID="cri-o://20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108" gracePeriod=30
Sep 16 12:09:11.075327 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.075299    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:09:11.118281 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118247    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e3c5739-a7c9-4e80-bab5-63637a1c4039-ca-trust-extracted\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118281    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-installation-pull-secrets\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-bound-sa-token\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118349    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-image-registry-private-configuration\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118487    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-certificates\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118511    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-trusted-ca\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118563    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsrd\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-kube-api-access-6wsrd\") pod \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\" (UID: \"4e3c5739-a7c9-4e80-bab5-63637a1c4039\") "
Sep 16 12:09:11.118976 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.118922    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:11.119263 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.119232    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:11.121443 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.121411    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:11.121443 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.121419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:11.121586 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.121491    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:11.121708 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.121692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-kube-api-access-6wsrd" (OuterVolumeSpecName: "kube-api-access-6wsrd") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "kube-api-access-6wsrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:11.121746 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.121724    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:11.129413 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.129390    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3c5739-a7c9-4e80-bab5-63637a1c4039-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4e3c5739-a7c9-4e80-bab5-63637a1c4039" (UID: "4e3c5739-a7c9-4e80-bab5-63637a1c4039"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:09:11.219239 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219219    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-installation-pull-secrets\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.219239 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219238    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-bound-sa-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.219364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219248    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-tls\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.219364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219256    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e3c5739-a7c9-4e80-bab5-63637a1c4039-image-registry-private-configuration\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.219364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219265    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-registry-certificates\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.219364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219273    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c5739-a7c9-4e80-bab5-63637a1c4039-trusted-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.219364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219281    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsrd\" (UniqueName: \"kubernetes.io/projected/4e3c5739-a7c9-4e80-bab5-63637a1c4039-kube-api-access-6wsrd\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.219364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.219289    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e3c5739-a7c9-4e80-bab5-63637a1c4039-ca-trust-extracted\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:11.421701 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.421635    2560 generic.go:356] "Generic (PLEG): container finished" podID="4e3c5739-a7c9-4e80-bab5-63637a1c4039" containerID="20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108" exitCode=0
Sep 16 12:09:11.421701 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.421686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" event={"ID":"4e3c5739-a7c9-4e80-bab5-63637a1c4039","Type":"ContainerDied","Data":"20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108"}
Sep 16 12:09:11.421701 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.421691    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw"
Sep 16 12:09:11.421894 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.421748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85b65544b8-8vmkw" event={"ID":"4e3c5739-a7c9-4e80-bab5-63637a1c4039","Type":"ContainerDied","Data":"550b427c89ff4830b7037e40d193eaad6cfcc2295c223be1a484129ce51ecb64"}
Sep 16 12:09:11.421894 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.421768    2560 scope.go:117] "RemoveContainer" containerID="20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108"
Sep 16 12:09:11.430997 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.430978    2560 scope.go:117] "RemoveContainer" containerID="20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108"
Sep 16 12:09:11.431304 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:09:11.431284    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108\": container with ID starting with 20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108 not found: ID does not exist" containerID="20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108"
Sep 16 12:09:11.431405 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.431313    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108"} err="failed to get container status \"20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108\": rpc error: code = NotFound desc = could not find container \"20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108\": container with ID starting with 20158a44b762b266f9c19d4989dbf055e17dd83949c9204383ee532325903108 not found: ID does not exist"
Sep 16 12:09:11.441550 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.441521    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85b65544b8-8vmkw"]
Sep 16 12:09:11.444473 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:11.444452    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85b65544b8-8vmkw"]
Sep 16 12:09:12.323345 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:12.323316    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039" path="/var/lib/kubelet/pods/4e3c5739-a7c9-4e80-bab5-63637a1c4039/volumes"
Sep 16 12:09:27.118294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.118239    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b87455d7-pqbjl" podUID="dc981f02-9682-4560-a586-606b4fd46cb6" containerName="console" containerID="cri-o://64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d" gracePeriod=15
Sep 16 12:09:27.352209 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.352184    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:09:27.449484 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.449425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mnp9\" (UniqueName: \"kubernetes.io/projected/dc981f02-9682-4560-a586-606b4fd46cb6-kube-api-access-7mnp9\") pod \"dc981f02-9682-4560-a586-606b4fd46cb6\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") "
Sep 16 12:09:27.449484 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.449473    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-oauth-config\") pod \"dc981f02-9682-4560-a586-606b4fd46cb6\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") "
Sep 16 12:09:27.449648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.449510    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-console-config\") pod \"dc981f02-9682-4560-a586-606b4fd46cb6\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") "
Sep 16 12:09:27.449648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.449529    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-serving-cert\") pod \"dc981f02-9682-4560-a586-606b4fd46cb6\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") "
Sep 16 12:09:27.449648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.449549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-oauth-serving-cert\") pod \"dc981f02-9682-4560-a586-606b4fd46cb6\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") "
Sep 16 12:09:27.449648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.449587    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-service-ca\") pod \"dc981f02-9682-4560-a586-606b4fd46cb6\" (UID: \"dc981f02-9682-4560-a586-606b4fd46cb6\") "
Sep 16 12:09:27.450191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.450129    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-console-config" (OuterVolumeSpecName: "console-config") pod "dc981f02-9682-4560-a586-606b4fd46cb6" (UID: "dc981f02-9682-4560-a586-606b4fd46cb6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:27.450302 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.450240    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc981f02-9682-4560-a586-606b4fd46cb6" (UID: "dc981f02-9682-4560-a586-606b4fd46cb6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:27.450453 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.450428    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-oauth-serving-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:27.450453 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.450452    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-console-config\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:27.450585 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.450525    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc981f02-9682-4560-a586-606b4fd46cb6" (UID: "dc981f02-9682-4560-a586-606b4fd46cb6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:27.452091 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.452065    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc981f02-9682-4560-a586-606b4fd46cb6" (UID: "dc981f02-9682-4560-a586-606b4fd46cb6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:27.452421 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.452400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc981f02-9682-4560-a586-606b4fd46cb6-kube-api-access-7mnp9" (OuterVolumeSpecName: "kube-api-access-7mnp9") pod "dc981f02-9682-4560-a586-606b4fd46cb6" (UID: "dc981f02-9682-4560-a586-606b4fd46cb6"). InnerVolumeSpecName "kube-api-access-7mnp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:27.452489 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.452411    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc981f02-9682-4560-a586-606b4fd46cb6" (UID: "dc981f02-9682-4560-a586-606b4fd46cb6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:27.487475 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.487447    2560 generic.go:356] "Generic (PLEG): container finished" podID="a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24" containerID="a4be6a17899114ad759fc884fc92bdc1b8a97400d7710782e3c74d7ef0bab930" exitCode=0
Sep 16 12:09:27.487571 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.487523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kdfjs" event={"ID":"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24","Type":"ContainerDied","Data":"a4be6a17899114ad759fc884fc92bdc1b8a97400d7710782e3c74d7ef0bab930"}
Sep 16 12:09:27.487943 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.487925    2560 scope.go:117] "RemoveContainer" containerID="a4be6a17899114ad759fc884fc92bdc1b8a97400d7710782e3c74d7ef0bab930"
Sep 16 12:09:27.494430 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.494406    2560 generic.go:356] "Generic (PLEG): container finished" podID="dc981f02-9682-4560-a586-606b4fd46cb6" containerID="64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d" exitCode=0
Sep 16 12:09:27.494515 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.494447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b87455d7-pqbjl" event={"ID":"dc981f02-9682-4560-a586-606b4fd46cb6","Type":"ContainerDied","Data":"64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d"}
Sep 16 12:09:27.494515 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.494462    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b87455d7-pqbjl"
Sep 16 12:09:27.494634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.494520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b87455d7-pqbjl" event={"ID":"dc981f02-9682-4560-a586-606b4fd46cb6","Type":"ContainerDied","Data":"84d49ef94bca8473a54f8c07d5cbce9b20f7b3bba80725e54aea9f11fd0cabb2"}
Sep 16 12:09:27.494634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.494543    2560 scope.go:117] "RemoveContainer" containerID="64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d"
Sep 16 12:09:27.505983 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.505967    2560 scope.go:117] "RemoveContainer" containerID="64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d"
Sep 16 12:09:27.506219 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:09:27.506204    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d\": container with ID starting with 64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d not found: ID does not exist" containerID="64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d"
Sep 16 12:09:27.506273 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.506226    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d"} err="failed to get container status \"64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d\": rpc error: code = NotFound desc = could not find container \"64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d\": container with ID starting with 64d2a1a02e2626ea715b6fc82e30b16f869d1533d6dda447d980d3b46646f93d not found: ID does not exist"
Sep 16 12:09:27.519678 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.519655    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b87455d7-pqbjl"]
Sep 16 12:09:27.528089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.528069    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b87455d7-pqbjl"]
Sep 16 12:09:27.551730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.551703    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mnp9\" (UniqueName: \"kubernetes.io/projected/dc981f02-9682-4560-a586-606b4fd46cb6-kube-api-access-7mnp9\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:27.551852 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.551730    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-oauth-config\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:27.551852 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.551745    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc981f02-9682-4560-a586-606b4fd46cb6-console-serving-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:27.551852 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:27.551758    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc981f02-9682-4560-a586-606b4fd46cb6-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:09:28.323446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:28.323414    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc981f02-9682-4560-a586-606b4fd46cb6" path="/var/lib/kubelet/pods/dc981f02-9682-4560-a586-606b4fd46cb6/volumes"
Sep 16 12:09:28.501271 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:28.501229    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kdfjs" event={"ID":"a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24","Type":"ContainerStarted","Data":"52be2d6b955e6d239454fbe437b7dea3f39db7883442f2c40cea0bc4e6f0ec51"}
Sep 16 12:09:37.233620 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:37.233576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:09:37.236466 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:37.236429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360bde0-4498-4c93-85ca-0813afa0fc95-metrics-certs\") pod \"network-metrics-daemon-sdmtp\" (UID: \"3360bde0-4498-4c93-85ca-0813afa0fc95\") " pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:09:37.422306 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:37.422278    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kdnz\""
Sep 16 12:09:37.430332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:37.430313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdmtp"
Sep 16 12:09:37.702009 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:37.701984    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdmtp"]
Sep 16 12:09:37.703997 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:09:37.703960    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3360bde0_4498_4c93_85ca_0813afa0fc95.slice/crio-ff82af629675ebadd5868e9925a2781a48ec5638828e8acb7cfc27ee68611c32 WatchSource:0}: Error finding container ff82af629675ebadd5868e9925a2781a48ec5638828e8acb7cfc27ee68611c32: Status 404 returned error can't find the container with id ff82af629675ebadd5868e9925a2781a48ec5638828e8acb7cfc27ee68611c32
Sep 16 12:09:38.545552 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:38.545518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdmtp" event={"ID":"3360bde0-4498-4c93-85ca-0813afa0fc95","Type":"ContainerStarted","Data":"ff82af629675ebadd5868e9925a2781a48ec5638828e8acb7cfc27ee68611c32"}
Sep 16 12:09:39.551391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:39.551339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdmtp" event={"ID":"3360bde0-4498-4c93-85ca-0813afa0fc95","Type":"ContainerStarted","Data":"18233cf96764537fd8104df31588a75bb5fe244859c0fa5e041c964ab7b8bdea"}
Sep 16 12:09:39.551748 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:39.551427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdmtp" event={"ID":"3360bde0-4498-4c93-85ca-0813afa0fc95","Type":"ContainerStarted","Data":"4b87b76258d9a5f2b965dba00d4de442fe2094892781e933c98dcb190e4feff5"}
Sep 16 12:09:39.571052 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:39.571006    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sdmtp" podStartSLOduration=374.470976327 podStartE2EDuration="6m15.570991285s" totalImagesPullingTime="1.100014958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:39.568514985 +0000 UTC m=+375.703836957" watchObservedRunningTime="2026-09-16 12:09:39.570991285 +0000 UTC m=+375.706313256"
Sep 16 12:09:44.396821 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:44.396775    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:09:44.413039 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:44.413008    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:09:44.591119 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:09:44.591094    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:02.434089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.434044    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:10:02.434671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.434350    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="prometheus" containerID="cri-o://f196bc8211acb2aa0e0b4a0f7011e995076cbaf510dc84179fb30dd744914766" gracePeriod=600
Sep 16 12:10:02.434671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.434379    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ae20ccc023a062c5edb5a2e19d254883cee6af10a74e0112ed0266e7177a7202" gracePeriod=600
Sep 16 12:10:02.434671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.434392    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy" containerID="cri-o://3ad882dea5a29ad5b0b721f814262dcba3c040a0508c7a4d5810990cb08fff1a" gracePeriod=600
Sep 16 12:10:02.434671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.434423    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="thanos-sidecar" containerID="cri-o://080b68d656a79c7af68646f6db338f2405219bac3d6543d83b1d47d8e5075e66" gracePeriod=600
Sep 16 12:10:02.434671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.434442    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="config-reloader" containerID="cri-o://c813f9e5f3e2a27b418ed99381161bc80d048738fa92dcb4815ee5f6f4afa2dc" gracePeriod=600
Sep 16 12:10:02.434671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.434533    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-web" containerID="cri-o://984bfef77f43a104c2f3088e4b3427f23e576b97841c98e953bfe5322b5aecc1" gracePeriod=600
Sep 16 12:10:02.649104 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649062    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerID="ae20ccc023a062c5edb5a2e19d254883cee6af10a74e0112ed0266e7177a7202" exitCode=0
Sep 16 12:10:02.649104 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649101    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerID="3ad882dea5a29ad5b0b721f814262dcba3c040a0508c7a4d5810990cb08fff1a" exitCode=0
Sep 16 12:10:02.649104 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649112    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerID="080b68d656a79c7af68646f6db338f2405219bac3d6543d83b1d47d8e5075e66" exitCode=0
Sep 16 12:10:02.649316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649121    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerID="c813f9e5f3e2a27b418ed99381161bc80d048738fa92dcb4815ee5f6f4afa2dc" exitCode=0
Sep 16 12:10:02.649316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649129    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerID="f196bc8211acb2aa0e0b4a0f7011e995076cbaf510dc84179fb30dd744914766" exitCode=0
Sep 16 12:10:02.649316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"ae20ccc023a062c5edb5a2e19d254883cee6af10a74e0112ed0266e7177a7202"}
Sep 16 12:10:02.649316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"3ad882dea5a29ad5b0b721f814262dcba3c040a0508c7a4d5810990cb08fff1a"}
Sep 16 12:10:02.649316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"080b68d656a79c7af68646f6db338f2405219bac3d6543d83b1d47d8e5075e66"}
Sep 16 12:10:02.649316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"c813f9e5f3e2a27b418ed99381161bc80d048738fa92dcb4815ee5f6f4afa2dc"}
Sep 16 12:10:02.649316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:02.649297    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"f196bc8211acb2aa0e0b4a0f7011e995076cbaf510dc84179fb30dd744914766"}
Sep 16 12:10:03.658924 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.658892    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerID="984bfef77f43a104c2f3088e4b3427f23e576b97841c98e953bfe5322b5aecc1" exitCode=0
Sep 16 12:10:03.659257 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.658983    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"984bfef77f43a104c2f3088e4b3427f23e576b97841c98e953bfe5322b5aecc1"}
Sep 16 12:10:03.678604 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.678584    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:03.748168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748136    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-kube-rbac-proxy\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748168 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-1\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748186    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-grpc-tls\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748203    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-trusted-ca-bundle\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748219    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-metrics-client-ca\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748249    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-config-out\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748280    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-serving-certs-ca-bundle\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748306    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-web-config\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748339    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-metrics-client-certs\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748364    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-0\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-thanos-prometheus-http-client-file\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748415    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72sh7\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-kube-api-access-72sh7\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-2\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748486    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-db\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748520    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-kubelet-serving-ca-bundle\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748544    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-tls-assets\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748585    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-config\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748597    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748613    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748652    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748681    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748715    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:03.748777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.748771    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-tls\") pod \"eb645edd-ae24-436a-8dc6-f16ec7e05441\" (UID: \"eb645edd-ae24-436a-8dc6-f16ec7e05441\") "
Sep 16 12:10:03.749467 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.749030    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:03.749467 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.749220    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.749467 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.749239    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.749467 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.749257    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-metrics-client-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.749467 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.749272    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.750324 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.750291    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:03.751002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.750714    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:03.751107 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.751025    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:03.751548 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.751519    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:10:03.752572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.752545    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.753300 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.753266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.753431 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.753404    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.753960 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.753934    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-config-out" (OuterVolumeSpecName: "config-out") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:10:03.754120 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.754096    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.754422 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.754402    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.754545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.754485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.754659 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.754631    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-kube-api-access-72sh7" (OuterVolumeSpecName: "kube-api-access-72sh7") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "kube-api-access-72sh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:03.754861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.754840    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.755388 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.755369    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:03.755767 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.755732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-config" (OuterVolumeSpecName: "config") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.764909 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.764854    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-web-config" (OuterVolumeSpecName: "web-config") pod "eb645edd-ae24-436a-8dc6-f16ec7e05441" (UID: "eb645edd-ae24-436a-8dc6-f16ec7e05441"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:03.850493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850472    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-db\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850494    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850504    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-tls-assets\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850513    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-config\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850521    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850529    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850538    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850547    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-kube-rbac-proxy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850555    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-grpc-tls\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850563    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb645edd-ae24-436a-8dc6-f16ec7e05441-config-out\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850570    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850578    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-web-config\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850585    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-secret-metrics-client-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850593    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb645edd-ae24-436a-8dc6-f16ec7e05441-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850602    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb645edd-ae24-436a-8dc6-f16ec7e05441-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:03.850648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:03.850609    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72sh7\" (UniqueName: \"kubernetes.io/projected/eb645edd-ae24-436a-8dc6-f16ec7e05441-kube-api-access-72sh7\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:10:04.666124 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.666095    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.666532 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.666093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb645edd-ae24-436a-8dc6-f16ec7e05441","Type":"ContainerDied","Data":"96abde73ca939d1171a8452a5669685e31518fdfeb44bea56bf474055608440d"}
Sep 16 12:10:04.666532 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.666269    2560 scope.go:117] "RemoveContainer" containerID="ae20ccc023a062c5edb5a2e19d254883cee6af10a74e0112ed0266e7177a7202"
Sep 16 12:10:04.673897 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.673880    2560 scope.go:117] "RemoveContainer" containerID="3ad882dea5a29ad5b0b721f814262dcba3c040a0508c7a4d5810990cb08fff1a"
Sep 16 12:10:04.680579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.680560    2560 scope.go:117] "RemoveContainer" containerID="984bfef77f43a104c2f3088e4b3427f23e576b97841c98e953bfe5322b5aecc1"
Sep 16 12:10:04.687167 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.687146    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:10:04.687930 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.687915    2560 scope.go:117] "RemoveContainer" containerID="080b68d656a79c7af68646f6db338f2405219bac3d6543d83b1d47d8e5075e66"
Sep 16 12:10:04.691376 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.691353    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:10:04.694766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.694751    2560 scope.go:117] "RemoveContainer" containerID="c813f9e5f3e2a27b418ed99381161bc80d048738fa92dcb4815ee5f6f4afa2dc"
Sep 16 12:10:04.701505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.701481    2560 scope.go:117] "RemoveContainer" containerID="f196bc8211acb2aa0e0b4a0f7011e995076cbaf510dc84179fb30dd744914766"
Sep 16 12:10:04.708504 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.708485    2560 scope.go:117] "RemoveContainer" containerID="d528a3cb05e8bbe31ce6e34347c71417a8b8debed20de5891b31e1db83f8c4bc"
Sep 16 12:10:04.718546 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718528    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:10:04.718837 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718816    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="config-reloader"
Sep 16 12:10:04.718837 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718832    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="config-reloader"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718841    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-web"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718846    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-web"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718853    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="thanos-sidecar"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718858    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="thanos-sidecar"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718866    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039" containerName="registry"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718872    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039" containerName="registry"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718877    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-thanos"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718882    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-thanos"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718890    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc981f02-9682-4560-a586-606b4fd46cb6" containerName="console"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718894    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc981f02-9682-4560-a586-606b4fd46cb6" containerName="console"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718902    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="init-config-reloader"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718907    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="init-config-reloader"
Sep 16 12:10:04.718910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718912    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="prometheus"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718917    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="prometheus"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718923    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718928    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718976    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="thanos-sidecar"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718982    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-web"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718987    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3c5739-a7c9-4e80-bab5-63637a1c4039" containerName="registry"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718994    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy-thanos"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.718999    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc981f02-9682-4560-a586-606b4fd46cb6" containerName="console"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.719005    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="config-reloader"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.719010    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="kube-rbac-proxy"
Sep 16 12:10:04.719279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.719015    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" containerName="prometheus"
Sep 16 12:10:04.722658 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.722640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.725366 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.725346    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1jbh1uj4b2m4f\""
Sep 16 12:10:04.725464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.725366    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 12:10:04.725464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.725386    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 12:10:04.725464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.725388    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 12:10:04.725729 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.725528    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 12:10:04.725835 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.725758    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 12:10:04.727972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.726320    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 12:10:04.727972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.726594    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 12:10:04.727972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.727261    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 12:10:04.727972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.727549    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:10:04.727972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.727759    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 12:10:04.728310 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.728077    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dw2jk\""
Sep 16 12:10:04.728716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.728456    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 12:10:04.728716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.728455    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 12:10:04.728716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.728542    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 12:10:04.730005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.729982    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 12:10:04.737154 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.737133    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 12:10:04.739602 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.739580    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:10:04.756567 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756663 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756663 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756663 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756764 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756764 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756764 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51206dfa-8638-4f66-ac3c-dfdbd22ff570-config-out\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756892 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756892 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58q2\" (UniqueName: \"kubernetes.io/projected/51206dfa-8638-4f66-ac3c-dfdbd22ff570-kube-api-access-m58q2\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756892 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51206dfa-8638-4f66-ac3c-dfdbd22ff570-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756930    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.756986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.757130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.756994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.757130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.757010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.757130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.757028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.757130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.757045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.757130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.757058    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-web-config\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.757130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.757072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-config\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.857644 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.857644 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.857831 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.857831 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.857951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.857951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-web-config\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.857951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-config\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.857988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51206dfa-8638-4f66-ac3c-dfdbd22ff570-config-out\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58q2\" (UniqueName: \"kubernetes.io/projected/51206dfa-8638-4f66-ac3c-dfdbd22ff570-kube-api-access-m58q2\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51206dfa-8638-4f66-ac3c-dfdbd22ff570-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.858428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.860100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.860100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.858973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.860100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.859304    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.860100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.859607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.860348 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.860298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.860535 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.860508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.861079 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.861032    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-web-config\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.861325 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.861305    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.862464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.862433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.862700 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.862680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.863306 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.863286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.863485 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.863464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.863688 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.863671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51206dfa-8638-4f66-ac3c-dfdbd22ff570-config-out\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.863905 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.863884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51206dfa-8638-4f66-ac3c-dfdbd22ff570-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.863964 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.863919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51206dfa-8638-4f66-ac3c-dfdbd22ff570-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.864205 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.864190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.864243 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.864219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-config\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.865088 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.865073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51206dfa-8638-4f66-ac3c-dfdbd22ff570-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:04.866253 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:04.866238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58q2\" (UniqueName: \"kubernetes.io/projected/51206dfa-8638-4f66-ac3c-dfdbd22ff570-kube-api-access-m58q2\") pod \"prometheus-k8s-0\" (UID: \"51206dfa-8638-4f66-ac3c-dfdbd22ff570\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:05.036566 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:05.036495    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:05.314712 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:05.314681    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:10:05.315774 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:10:05.315747    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51206dfa_8638_4f66_ac3c_dfdbd22ff570.slice/crio-f3005229d5b511f36ea3a78df4471baeba037356ee679cfbe9cdcafeb487d449 WatchSource:0}: Error finding container f3005229d5b511f36ea3a78df4471baeba037356ee679cfbe9cdcafeb487d449: Status 404 returned error can't find the container with id f3005229d5b511f36ea3a78df4471baeba037356ee679cfbe9cdcafeb487d449
Sep 16 12:10:05.672546 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:05.672510    2560 generic.go:356] "Generic (PLEG): container finished" podID="51206dfa-8638-4f66-ac3c-dfdbd22ff570" containerID="abb96f6d8755a84bf0238be281a8f5cdc17335a38bbbd34d01e93beabf961be0" exitCode=0
Sep 16 12:10:05.672919 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:05.672572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerDied","Data":"abb96f6d8755a84bf0238be281a8f5cdc17335a38bbbd34d01e93beabf961be0"}
Sep 16 12:10:05.672919 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:05.672639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerStarted","Data":"f3005229d5b511f36ea3a78df4471baeba037356ee679cfbe9cdcafeb487d449"}
Sep 16 12:10:06.324057 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.323991    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb645edd-ae24-436a-8dc6-f16ec7e05441" path="/var/lib/kubelet/pods/eb645edd-ae24-436a-8dc6-f16ec7e05441/volumes"
Sep 16 12:10:06.680625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.680595    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerStarted","Data":"18c3ac424596eaf9015eb015b102d687b9941e7211e675a1b98ee3361b701cd7"}
Sep 16 12:10:06.681054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.680667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerStarted","Data":"3393910187ea117fee40605ec3b11af3d9b2910c39f7f68c3952363959ccae3f"}
Sep 16 12:10:06.681054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.680721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerStarted","Data":"11b79de8d1c6d4d75da752b62f8336e64bd62de74156d3febe47a8f2dbb22f2f"}
Sep 16 12:10:06.681054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.680763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerStarted","Data":"304b05b5050a119fdeb510079c7693b5dcc1934a06071b599124b99fce034baa"}
Sep 16 12:10:06.681054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.680827    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerStarted","Data":"740ff1f3314d4e64a084490e6a4d9e520173c2cdd916e9a47d5244b3c994737b"}
Sep 16 12:10:06.681054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.680879    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51206dfa-8638-4f66-ac3c-dfdbd22ff570","Type":"ContainerStarted","Data":"d17537da73bac6063149886f229c0fe8c188eb9959437797e32abaae7ce4ac2f"}
Sep 16 12:10:06.710283 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:06.710237    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.690802958 podStartE2EDuration="2.710223487s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.420529ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:06.707528765 +0000 UTC m=+402.842850735" watchObservedRunningTime="2026-09-16 12:10:06.710223487 +0000 UTC m=+402.845545458"
Sep 16 12:10:07.131985 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:10:07.131949    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh" podUID="c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90"
Sep 16 12:10:07.684153 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:07.684121    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:10:09.697065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:09.697031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:10:09.699660 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:09.699633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j52wh\" (UID: \"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:10:09.787643 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:09.787614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t4tt2\""
Sep 16 12:10:09.795598 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:09.795577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"
Sep 16 12:10:10.037028 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:10.036958    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:10:10.067291 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:10.067181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh"]
Sep 16 12:10:10.069832 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:10:10.069803    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8aa3fd1_fdb8_4f06_82f4_01dfe5c84f90.slice/crio-ecd5649004d2f2a01d3bf1d4b72b47c1074e393be7eaeea1d15244fb3b53c00a WatchSource:0}: Error finding container ecd5649004d2f2a01d3bf1d4b72b47c1074e393be7eaeea1d15244fb3b53c00a: Status 404 returned error can't find the container with id ecd5649004d2f2a01d3bf1d4b72b47c1074e393be7eaeea1d15244fb3b53c00a
Sep 16 12:10:10.696584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:10.696547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh" event={"ID":"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90","Type":"ContainerStarted","Data":"ecd5649004d2f2a01d3bf1d4b72b47c1074e393be7eaeea1d15244fb3b53c00a"}
Sep 16 12:10:11.706018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:11.705986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh" event={"ID":"c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90","Type":"ContainerStarted","Data":"55c04f91bc3b4f7b8f796ff5904ab950557c7fd4696d5fdae286ec9e93f5d1c7"}
Sep 16 12:10:11.721378 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:10:11.721332    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j52wh" podStartSLOduration=393.268444707 podStartE2EDuration="6m34.721317073s" totalImagesPullingTime="1.452872366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:11.720446836 +0000 UTC m=+407.855768808" watchObservedRunningTime="2026-09-16 12:10:11.721317073 +0000 UTC m=+407.856639044"
Sep 16 12:11:05.037214 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:11:05.037177    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:11:05.054494 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:11:05.054469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:11:05.932214 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:11:05.932185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:12:09.935573 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.935539    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"]
Sep 16 12:12:09.938534 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.938517    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:09.942693 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.942667    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 12:12:09.942819 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.942770    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:12:09.942819 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.942801    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:12:09.942819 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.942808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 12:12:09.942982 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.942831    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 12:12:09.943160 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.943144    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rhnbv\""
Sep 16 12:12:09.950595 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:09.950568    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"]
Sep 16 12:12:10.083881 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.083846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b5848b3-575a-4d5d-8562-1ce29a0182b8-manager-config\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.083881 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.083876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b5848b3-575a-4d5d-8562-1ce29a0182b8-metrics-cert\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.084068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.083917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66fp\" (UniqueName: \"kubernetes.io/projected/9b5848b3-575a-4d5d-8562-1ce29a0182b8-kube-api-access-x66fp\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.084068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.083976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b5848b3-575a-4d5d-8562-1ce29a0182b8-cert\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.185099 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.185055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b5848b3-575a-4d5d-8562-1ce29a0182b8-manager-config\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.185099 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.185101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b5848b3-575a-4d5d-8562-1ce29a0182b8-metrics-cert\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.185352 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.185155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66fp\" (UniqueName: \"kubernetes.io/projected/9b5848b3-575a-4d5d-8562-1ce29a0182b8-kube-api-access-x66fp\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.185352 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.185183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b5848b3-575a-4d5d-8562-1ce29a0182b8-cert\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.185885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.185829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b5848b3-575a-4d5d-8562-1ce29a0182b8-manager-config\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.187727 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.187708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b5848b3-575a-4d5d-8562-1ce29a0182b8-cert\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.187846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.187775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b5848b3-575a-4d5d-8562-1ce29a0182b8-metrics-cert\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.223546 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.223515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66fp\" (UniqueName: \"kubernetes.io/projected/9b5848b3-575a-4d5d-8562-1ce29a0182b8-kube-api-access-x66fp\") pod \"lws-controller-manager-74d55495b7-hrcxp\" (UID: \"9b5848b3-575a-4d5d-8562-1ce29a0182b8\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.248494 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.248465    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:10.519952 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:10.519819    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"]
Sep 16 12:12:10.522473 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:12:10.522442    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5848b3_575a_4d5d_8562_1ce29a0182b8.slice/crio-52b7da47a3d2feb217b90b88d5b6431f6b64d497b00725a60a16971407f933ff WatchSource:0}: Error finding container 52b7da47a3d2feb217b90b88d5b6431f6b64d497b00725a60a16971407f933ff: Status 404 returned error can't find the container with id 52b7da47a3d2feb217b90b88d5b6431f6b64d497b00725a60a16971407f933ff
Sep 16 12:12:11.175695 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:11.175657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp" event={"ID":"9b5848b3-575a-4d5d-8562-1ce29a0182b8","Type":"ContainerStarted","Data":"52b7da47a3d2feb217b90b88d5b6431f6b64d497b00725a60a16971407f933ff"}
Sep 16 12:12:13.185704 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:13.185622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp" event={"ID":"9b5848b3-575a-4d5d-8562-1ce29a0182b8","Type":"ContainerStarted","Data":"ccf2933769ed2096c8c2b5bfde9ad4bb2ab8403c70d6afe9320e8842db3b85b3"}
Sep 16 12:12:13.186079 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:13.185746    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:13.205240 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:13.204765    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp" podStartSLOduration=1.7975526259999999 podStartE2EDuration="4.204745512s" totalImagesPullingTime="2.407192886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:13.204046367 +0000 UTC m=+529.339368340" watchObservedRunningTime="2026-09-16 12:12:13.204745512 +0000 UTC m=+529.340067482"
Sep 16 12:12:14.424771 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.424741    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"]
Sep 16 12:12:14.428196 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.428170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.438347 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.438326    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:12:14.438456 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.438357    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:12:14.438521 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.438481    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nps9s\""
Sep 16 12:12:14.438583 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.438537    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:12:14.438750 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.438734    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:12:14.457119 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.457093    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"]
Sep 16 12:12:14.522944 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.522911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92t7f\" (UniqueName: \"kubernetes.io/projected/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-kube-api-access-92t7f\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.523117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.522964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.523117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.522994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.523117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.523049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.523117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.523084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.624182 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.624137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.624362 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.624196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.624362 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.624222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92t7f\" (UniqueName: \"kubernetes.io/projected/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-kube-api-access-92t7f\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.624362 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.624252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.624362 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.624283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.624643 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.624613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.626859 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.626832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.626970 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.626942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.627067 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.627047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.635203 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.635179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92t7f\" (UniqueName: \"kubernetes.io/projected/b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1-kube-api-access-92t7f\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp\" (UID: \"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:14.738447 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:14.738363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:15.032382 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:15.032312    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"]
Sep 16 12:12:15.035260 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:12:15.035231    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36796c9_c3a4_4d67_a94f_bd8e0f9fdbd1.slice/crio-46c915d7dcfe933430e52f29bef0d7d97465d6b0a3f90804d2ceff40a1a5afac WatchSource:0}: Error finding container 46c915d7dcfe933430e52f29bef0d7d97465d6b0a3f90804d2ceff40a1a5afac: Status 404 returned error can't find the container with id 46c915d7dcfe933430e52f29bef0d7d97465d6b0a3f90804d2ceff40a1a5afac
Sep 16 12:12:15.195327 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:15.195294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp" event={"ID":"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1","Type":"ContainerStarted","Data":"46c915d7dcfe933430e52f29bef0d7d97465d6b0a3f90804d2ceff40a1a5afac"}
Sep 16 12:12:18.211102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:18.211032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp" event={"ID":"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1","Type":"ContainerStarted","Data":"dfcfb886475b78a4a29f952e1d76929ddcc9ba368473e4b8676c3872f5ff3d1f"}
Sep 16 12:12:19.217109 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:19.217074    2560 generic.go:356] "Generic (PLEG): container finished" podID="b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1" containerID="dfcfb886475b78a4a29f952e1d76929ddcc9ba368473e4b8676c3872f5ff3d1f" exitCode=0
Sep 16 12:12:19.217437 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:19.217155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp" event={"ID":"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1","Type":"ContainerDied","Data":"dfcfb886475b78a4a29f952e1d76929ddcc9ba368473e4b8676c3872f5ff3d1f"}
Sep 16 12:12:20.224511 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:20.224468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp" event={"ID":"b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1","Type":"ContainerStarted","Data":"68c20008bc43da2fd9a248e8f1339224bc0adc6341eecfbbf7f1b129f809f844"}
Sep 16 12:12:20.224998 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:20.224580    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:20.280320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:20.280267    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp" podStartSLOduration=1.946229206 podStartE2EDuration="6.280251468s" totalImagesPullingTime="3.252130887s" totalInitContainerRuntime="1.081891375s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:20.278906411 +0000 UTC m=+536.414228382" watchObservedRunningTime="2026-09-16 12:12:20.280251468 +0000 UTC m=+536.415573438"
Sep 16 12:12:24.193041 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:24.193005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-hrcxp"
Sep 16 12:12:31.231033 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:31.231005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp"
Sep 16 12:12:44.196428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.196391    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"]
Sep 16 12:12:44.209124 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.209098    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.210887 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.210849    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"]
Sep 16 12:12:44.212315 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.212291    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t28rh\""
Sep 16 12:12:44.212555 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.212535    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 12:12:44.213485 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.213466    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 12:12:44.261494 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.261459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5sjm\" (UniqueName: \"kubernetes.io/projected/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-kube-api-access-x5sjm\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.261624 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.261499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tmp\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.261624 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.261566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.362001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.361967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.362176 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.362119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sjm\" (UniqueName: \"kubernetes.io/projected/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-kube-api-access-x5sjm\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.362176 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:12:44.362151    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 12:12:44.362287 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:12:44.362227    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tls-certs podName:df1ff7b3-88ce-46cf-bdbe-f45c472f0458 nodeName:}" failed. No retries permitted until 2026-09-16 12:12:44.862206028 +0000 UTC m=+560.997527982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tls-certs") pod "kube-auth-proxy-6f67fb6dbc-96rxq" (UID: "df1ff7b3-88ce-46cf-bdbe-f45c472f0458") : secret "kube-auth-proxy-tls" not found
Sep 16 12:12:44.362357 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.362155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tmp\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.364512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.364482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tmp\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.371555 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.371532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sjm\" (UniqueName: \"kubernetes.io/projected/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-kube-api-access-x5sjm\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.867149 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.867118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:44.869870 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:44.869838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ff7b3-88ce-46cf-bdbe-f45c472f0458-tls-certs\") pod \"kube-auth-proxy-6f67fb6dbc-96rxq\" (UID: \"df1ff7b3-88ce-46cf-bdbe-f45c472f0458\") " pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:45.123320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:45.123233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"
Sep 16 12:12:45.405840 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:45.405818    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq"]
Sep 16 12:12:45.412106 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:12:45.412041    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1ff7b3_88ce_46cf_bdbe_f45c472f0458.slice/crio-df7ba009a8baa5b0ff2b81a4fa37fdbf814c3786ecb35db080cd249f77eed3d7 WatchSource:0}: Error finding container df7ba009a8baa5b0ff2b81a4fa37fdbf814c3786ecb35db080cd249f77eed3d7: Status 404 returned error can't find the container with id df7ba009a8baa5b0ff2b81a4fa37fdbf814c3786ecb35db080cd249f77eed3d7
Sep 16 12:12:46.346110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:46.346063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq" event={"ID":"df1ff7b3-88ce-46cf-bdbe-f45c472f0458","Type":"ContainerStarted","Data":"df7ba009a8baa5b0ff2b81a4fa37fdbf814c3786ecb35db080cd249f77eed3d7"}
Sep 16 12:12:48.357123 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:48.357084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq" event={"ID":"df1ff7b3-88ce-46cf-bdbe-f45c472f0458","Type":"ContainerStarted","Data":"5b8e0ec2549725ec1375b8883de30b54636dfbf4d914f7d76923d5691222137a"}
Sep 16 12:12:48.372974 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:12:48.372920    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f67fb6dbc-96rxq" podStartSLOduration=1.652413114 podStartE2EDuration="4.372906074s" totalImagesPullingTime="2.72049296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:48.371768281 +0000 UTC m=+564.507090254" watchObservedRunningTime="2026-09-16 12:12:48.372906074 +0000 UTC m=+564.508228042"
Sep 16 12:13:24.275725 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:13:24.275693    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:13:24.276303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:13:24.275950    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:14:08.429930 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.429893    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"]
Sep 16 12:14:08.433491 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.433475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:08.436328 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.436303    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mbkvc\""
Sep 16 12:14:08.436447 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.436306    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:14:08.436447 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.436315    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:14:08.443998 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.443976    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"]
Sep 16 12:14:08.487073 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.487045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fe16f9-a753-4a8a-8556-377d04756d14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:08.487190 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.487079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb29t\" (UniqueName: \"kubernetes.io/projected/f2fe16f9-a753-4a8a-8556-377d04756d14-kube-api-access-sb29t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:08.587966 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.587935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fe16f9-a753-4a8a-8556-377d04756d14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:08.587966 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.587971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb29t\" (UniqueName: \"kubernetes.io/projected/f2fe16f9-a753-4a8a-8556-377d04756d14-kube-api-access-sb29t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:08.588343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.588323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fe16f9-a753-4a8a-8556-377d04756d14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:08.598301 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.598273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb29t\" (UniqueName: \"kubernetes.io/projected/f2fe16f9-a753-4a8a-8556-377d04756d14-kube-api-access-sb29t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:08.744382 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:08.744287    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:09.025179 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:09.025145    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"]
Sep 16 12:14:09.028035 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:14:09.027998    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fe16f9_a753_4a8a_8556_377d04756d14.slice/crio-808c6016a8ea3c03138ea6c2bca965970b082568b9f7f0fea554d85ca6aec2b7 WatchSource:0}: Error finding container 808c6016a8ea3c03138ea6c2bca965970b082568b9f7f0fea554d85ca6aec2b7: Status 404 returned error can't find the container with id 808c6016a8ea3c03138ea6c2bca965970b082568b9f7f0fea554d85ca6aec2b7
Sep 16 12:14:09.030966 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:09.030949    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:14:09.716758 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:09.716722    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr" event={"ID":"f2fe16f9-a753-4a8a-8556-377d04756d14","Type":"ContainerStarted","Data":"808c6016a8ea3c03138ea6c2bca965970b082568b9f7f0fea554d85ca6aec2b7"}
Sep 16 12:14:36.835753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:36.835720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr" event={"ID":"f2fe16f9-a753-4a8a-8556-377d04756d14","Type":"ContainerStarted","Data":"747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf"}
Sep 16 12:14:36.836163 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:36.835852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:36.856327 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:36.856274    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr" podStartSLOduration=1.634511862 podStartE2EDuration="28.856255978s" totalImagesPullingTime="27.221744116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:36.85468818 +0000 UTC m=+672.990010172" watchObservedRunningTime="2026-09-16 12:14:36.856255978 +0000 UTC m=+672.991577945"
Sep 16 12:14:47.842018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:47.841988    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:49.977144 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:49.977111    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"]
Sep 16 12:14:49.977548 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:49.977278    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr" podUID="f2fe16f9-a753-4a8a-8556-377d04756d14" containerName="manager" containerID="cri-o://747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf" gracePeriod=2
Sep 16 12:14:49.986208 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:49.986182    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"]
Sep 16 12:14:49.995213 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:14:49.995171    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f2fe16f9-a753-4a8a-8556-377d04756d14\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-79jbr\": pods \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:50.005462 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.005439    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"]
Sep 16 12:14:50.005903 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.005883    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe16f9-a753-4a8a-8556-377d04756d14" containerName="manager"
Sep 16 12:14:50.005903 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.005904    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe16f9-a753-4a8a-8556-377d04756d14" containerName="manager"
Sep 16 12:14:50.006049 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.006014    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe16f9-a753-4a8a-8556-377d04756d14" containerName="manager"
Sep 16 12:14:50.008961 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.008942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.012610 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:14:50.012580    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" is forbidden: User \"system:node:ip-10-0-135-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-242.ec2.internal' and this object" podUID="f2fe16f9-a753-4a8a-8556-377d04756d14" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:50.029965 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.029926    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"]
Sep 16 12:14:50.123055 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.123023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvq5\" (UniqueName: \"kubernetes.io/projected/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-kube-api-access-2vvq5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngr6c\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.123191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.123066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngr6c\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.207076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.207052    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:50.209562 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:14:50.209536    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-79jbr\" is forbidden: User \"system:node:ip-10-0-135-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-242.ec2.internal' and this object" podUID="f2fe16f9-a753-4a8a-8556-377d04756d14" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:50.223941 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.223918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvq5\" (UniqueName: \"kubernetes.io/projected/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-kube-api-access-2vvq5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngr6c\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.224050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.223968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngr6c\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.224309 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.224292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngr6c\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.235536 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.235488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvq5\" (UniqueName: \"kubernetes.io/projected/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-kube-api-access-2vvq5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ngr6c\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.324259 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.324241    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb29t\" (UniqueName: \"kubernetes.io/projected/f2fe16f9-a753-4a8a-8556-377d04756d14-kube-api-access-sb29t\") pod \"f2fe16f9-a753-4a8a-8556-377d04756d14\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") "
Sep 16 12:14:50.324361 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.324321    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fe16f9-a753-4a8a-8556-377d04756d14-extensions-socket-volume\") pod \"f2fe16f9-a753-4a8a-8556-377d04756d14\" (UID: \"f2fe16f9-a753-4a8a-8556-377d04756d14\") "
Sep 16 12:14:50.324762 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.324740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe16f9-a753-4a8a-8556-377d04756d14-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f2fe16f9-a753-4a8a-8556-377d04756d14" (UID: "f2fe16f9-a753-4a8a-8556-377d04756d14"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:14:50.326461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.326441    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe16f9-a753-4a8a-8556-377d04756d14-kube-api-access-sb29t" (OuterVolumeSpecName: "kube-api-access-sb29t") pod "f2fe16f9-a753-4a8a-8556-377d04756d14" (UID: "f2fe16f9-a753-4a8a-8556-377d04756d14"). InnerVolumeSpecName "kube-api-access-sb29t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:14:50.355611 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.355588    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.425469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.425385    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb29t\" (UniqueName: \"kubernetes.io/projected/f2fe16f9-a753-4a8a-8556-377d04756d14-kube-api-access-sb29t\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:14:50.425469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.425426    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2fe16f9-a753-4a8a-8556-377d04756d14-extensions-socket-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:14:50.634149 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.634125    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"]
Sep 16 12:14:50.636640 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:14:50.636616    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac3f030_ef21_4396_bac2_d5f7e6f77fcc.slice/crio-485e8cc2a0cfdd7493b687299a0c9cbc32dc41ee8d298578c53fd5adc9977cf2 WatchSource:0}: Error finding container 485e8cc2a0cfdd7493b687299a0c9cbc32dc41ee8d298578c53fd5adc9977cf2: Status 404 returned error can't find the container with id 485e8cc2a0cfdd7493b687299a0c9cbc32dc41ee8d298578c53fd5adc9977cf2
Sep 16 12:14:50.897114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.897025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c" event={"ID":"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc","Type":"ContainerStarted","Data":"19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a"}
Sep 16 12:14:50.897114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.897089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c" event={"ID":"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc","Type":"ContainerStarted","Data":"485e8cc2a0cfdd7493b687299a0c9cbc32dc41ee8d298578c53fd5adc9977cf2"}
Sep 16 12:14:50.897302 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.897117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:14:50.898457 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.898428    2560 generic.go:356] "Generic (PLEG): container finished" podID="f2fe16f9-a753-4a8a-8556-377d04756d14" containerID="747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf" exitCode=0
Sep 16 12:14:50.898566 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.898480    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-79jbr"
Sep 16 12:14:50.898566 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.898499    2560 scope.go:117] "RemoveContainer" containerID="747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf"
Sep 16 12:14:50.907439 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.907420    2560 scope.go:117] "RemoveContainer" containerID="747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf"
Sep 16 12:14:50.907702 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:14:50.907681    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf\": container with ID starting with 747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf not found: ID does not exist" containerID="747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf"
Sep 16 12:14:50.907766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.907714    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf"} err="failed to get container status \"747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf\": rpc error: code = NotFound desc = could not find container \"747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf\": container with ID starting with 747e8aa10adc93e0dd60af7ae37d847bdc4466b0b0a137d0e8b47f4aef96ebcf not found: ID does not exist"
Sep 16 12:14:50.918219 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:50.918175    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c" podStartSLOduration=1.9181619049999998 podStartE2EDuration="1.918161905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:50.916755667 +0000 UTC m=+687.052077648" watchObservedRunningTime="2026-09-16 12:14:50.918161905 +0000 UTC m=+687.053483876"
Sep 16 12:14:52.323591 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:14:52.323558    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fe16f9-a753-4a8a-8556-377d04756d14" path="/var/lib/kubelet/pods/f2fe16f9-a753-4a8a-8556-377d04756d14/volumes"
Sep 16 12:15:01.905670 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:01.905639    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:15:05.359539 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.359508    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"]
Sep 16 12:15:05.359909 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.359697    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c" podUID="6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" containerName="manager" containerID="cri-o://19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a" gracePeriod=10
Sep 16 12:15:05.571447 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.571419    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"]
Sep 16 12:15:05.590779 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.590747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"]
Sep 16 12:15:05.590933 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.590866    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.630069 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.630044    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:15:05.740822 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.740772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vvq5\" (UniqueName: \"kubernetes.io/projected/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-kube-api-access-2vvq5\") pod \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") "
Sep 16 12:15:05.740939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.740827    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-extensions-socket-volume\") pod \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\" (UID: \"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc\") "
Sep 16 12:15:05.741030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.740998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32025949-7bc3-437e-b7d5-728153f2a8a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24cm7\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.741141 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.741041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkm8\" (UniqueName: \"kubernetes.io/projected/32025949-7bc3-437e-b7d5-728153f2a8a9-kube-api-access-5bkm8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24cm7\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.741223 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.741206    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" (UID: "6ac3f030-ef21-4396-bac2-d5f7e6f77fcc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:05.742930 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.742908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-kube-api-access-2vvq5" (OuterVolumeSpecName: "kube-api-access-2vvq5") pod "6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" (UID: "6ac3f030-ef21-4396-bac2-d5f7e6f77fcc"). InnerVolumeSpecName "kube-api-access-2vvq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:05.842356 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.842326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32025949-7bc3-437e-b7d5-728153f2a8a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24cm7\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.842470 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.842371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkm8\" (UniqueName: \"kubernetes.io/projected/32025949-7bc3-437e-b7d5-728153f2a8a9-kube-api-access-5bkm8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24cm7\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.842470 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.842432    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvq5\" (UniqueName: \"kubernetes.io/projected/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-kube-api-access-2vvq5\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:15:05.842470 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.842446    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc-extensions-socket-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:15:05.842730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.842706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32025949-7bc3-437e-b7d5-728153f2a8a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24cm7\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.850251 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.850227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkm8\" (UniqueName: \"kubernetes.io/projected/32025949-7bc3-437e-b7d5-728153f2a8a9-kube-api-access-5bkm8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-24cm7\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.902244 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.902192    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:05.964310 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.964280    2560 generic.go:356] "Generic (PLEG): container finished" podID="6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" containerID="19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a" exitCode=0
Sep 16 12:15:05.964458 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.964340    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"
Sep 16 12:15:05.964458 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.964376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c" event={"ID":"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc","Type":"ContainerDied","Data":"19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a"}
Sep 16 12:15:05.964458 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.964430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c" event={"ID":"6ac3f030-ef21-4396-bac2-d5f7e6f77fcc","Type":"ContainerDied","Data":"485e8cc2a0cfdd7493b687299a0c9cbc32dc41ee8d298578c53fd5adc9977cf2"}
Sep 16 12:15:05.964664 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.964460    2560 scope.go:117] "RemoveContainer" containerID="19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a"
Sep 16 12:15:05.975376 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.975358    2560 scope.go:117] "RemoveContainer" containerID="19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a"
Sep 16 12:15:05.975681 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:15:05.975662    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a\": container with ID starting with 19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a not found: ID does not exist" containerID="19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a"
Sep 16 12:15:05.975758 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.975690    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a"} err="failed to get container status \"19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a\": rpc error: code = NotFound desc = could not find container \"19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a\": container with ID starting with 19049b7880af745373fe9dae480bbc6aef45374a45bff5268eadf78267ac634a not found: ID does not exist"
Sep 16 12:15:05.989481 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.989453    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"]
Sep 16 12:15:05.994590 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:05.994547    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ngr6c"]
Sep 16 12:15:06.205094 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:06.205069    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"]
Sep 16 12:15:06.207756 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:15:06.207727    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32025949_7bc3_437e_b7d5_728153f2a8a9.slice/crio-0f57079991b20bc97dba1df70ec3208d1a52bf84b7aeff2ea2a93cf3bc87a9e8 WatchSource:0}: Error finding container 0f57079991b20bc97dba1df70ec3208d1a52bf84b7aeff2ea2a93cf3bc87a9e8: Status 404 returned error can't find the container with id 0f57079991b20bc97dba1df70ec3208d1a52bf84b7aeff2ea2a93cf3bc87a9e8
Sep 16 12:15:06.332994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:06.332969    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" path="/var/lib/kubelet/pods/6ac3f030-ef21-4396-bac2-d5f7e6f77fcc/volumes"
Sep 16 12:15:06.971042 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:06.971002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7" event={"ID":"32025949-7bc3-437e-b7d5-728153f2a8a9","Type":"ContainerStarted","Data":"ff5e5a13511dbf98e80b7770bafd689d28045fe3b8419374da8f3f45ca956bf2"}
Sep 16 12:15:06.971522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:06.971062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7" event={"ID":"32025949-7bc3-437e-b7d5-728153f2a8a9","Type":"ContainerStarted","Data":"0f57079991b20bc97dba1df70ec3208d1a52bf84b7aeff2ea2a93cf3bc87a9e8"}
Sep 16 12:15:06.971522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:06.971129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:06.998180 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:06.998136    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7" podStartSLOduration=1.998124053 podStartE2EDuration="1.998124053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:06.995356351 +0000 UTC m=+703.130678321" watchObservedRunningTime="2026-09-16 12:15:06.998124053 +0000 UTC m=+703.133446023"
Sep 16 12:15:17.979337 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:17.979267    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:15:36.087781 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.087748    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jmxft"]
Sep 16 12:15:36.088189 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.088161    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" containerName="manager"
Sep 16 12:15:36.088189 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.088179    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" containerName="manager"
Sep 16 12:15:36.088344 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.088325    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac3f030-ef21-4396-bac2-d5f7e6f77fcc" containerName="manager"
Sep 16 12:15:36.090977 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.090956    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft"
Sep 16 12:15:36.093462 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.093445    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-klzvv\""
Sep 16 12:15:36.098295 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.098088    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jmxft"]
Sep 16 12:15:36.173622 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.173595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4zv\" (UniqueName: \"kubernetes.io/projected/5e30e789-4ae6-4ad8-b067-8c49e92857d8-kube-api-access-vv4zv\") pod \"authorino-7bcfd8f9cd-jmxft\" (UID: \"5e30e789-4ae6-4ad8-b067-8c49e92857d8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft"
Sep 16 12:15:36.190779 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.190753    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-ms47b"]
Sep 16 12:15:36.192887 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.192867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ms47b"
Sep 16 12:15:36.198615 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.198594    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ms47b"]
Sep 16 12:15:36.274859 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.274834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4zv\" (UniqueName: \"kubernetes.io/projected/5e30e789-4ae6-4ad8-b067-8c49e92857d8-kube-api-access-vv4zv\") pod \"authorino-7bcfd8f9cd-jmxft\" (UID: \"5e30e789-4ae6-4ad8-b067-8c49e92857d8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft"
Sep 16 12:15:36.275002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.274920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkshh\" (UniqueName: \"kubernetes.io/projected/c40a86af-2c1d-45a6-a409-2a4bb1d34252-kube-api-access-lkshh\") pod \"authorino-59c974bd48-ms47b\" (UID: \"c40a86af-2c1d-45a6-a409-2a4bb1d34252\") " pod="kuadrant-system/authorino-59c974bd48-ms47b"
Sep 16 12:15:36.285024 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.284996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4zv\" (UniqueName: \"kubernetes.io/projected/5e30e789-4ae6-4ad8-b067-8c49e92857d8-kube-api-access-vv4zv\") pod \"authorino-7bcfd8f9cd-jmxft\" (UID: \"5e30e789-4ae6-4ad8-b067-8c49e92857d8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft"
Sep 16 12:15:36.376056 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.376032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkshh\" (UniqueName: \"kubernetes.io/projected/c40a86af-2c1d-45a6-a409-2a4bb1d34252-kube-api-access-lkshh\") pod \"authorino-59c974bd48-ms47b\" (UID: \"c40a86af-2c1d-45a6-a409-2a4bb1d34252\") " pod="kuadrant-system/authorino-59c974bd48-ms47b"
Sep 16 12:15:36.386455 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.386433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkshh\" (UniqueName: \"kubernetes.io/projected/c40a86af-2c1d-45a6-a409-2a4bb1d34252-kube-api-access-lkshh\") pod \"authorino-59c974bd48-ms47b\" (UID: \"c40a86af-2c1d-45a6-a409-2a4bb1d34252\") " pod="kuadrant-system/authorino-59c974bd48-ms47b"
Sep 16 12:15:36.413774 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.413753    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft"
Sep 16 12:15:36.502206 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.502177    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ms47b"
Sep 16 12:15:36.679817 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.679733    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jmxft"]
Sep 16 12:15:36.682637 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:15:36.682608    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e30e789_4ae6_4ad8_b067_8c49e92857d8.slice/crio-31d6e7ca8636d8d962c4ee23297eb8ee18f387ab5e8da7bf9d1cbd905f93e1af WatchSource:0}: Error finding container 31d6e7ca8636d8d962c4ee23297eb8ee18f387ab5e8da7bf9d1cbd905f93e1af: Status 404 returned error can't find the container with id 31d6e7ca8636d8d962c4ee23297eb8ee18f387ab5e8da7bf9d1cbd905f93e1af
Sep 16 12:15:36.769408 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:36.769384    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ms47b"]
Sep 16 12:15:36.771769 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:15:36.771747    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc40a86af_2c1d_45a6_a409_2a4bb1d34252.slice/crio-d661cd5d008e0a7529bc6db1c8aabe9d5bce0c6d38cebc716c0df011c4efdcba WatchSource:0}: Error finding container d661cd5d008e0a7529bc6db1c8aabe9d5bce0c6d38cebc716c0df011c4efdcba: Status 404 returned error can't find the container with id d661cd5d008e0a7529bc6db1c8aabe9d5bce0c6d38cebc716c0df011c4efdcba
Sep 16 12:15:37.112859 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:37.112826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft" event={"ID":"5e30e789-4ae6-4ad8-b067-8c49e92857d8","Type":"ContainerStarted","Data":"31d6e7ca8636d8d962c4ee23297eb8ee18f387ab5e8da7bf9d1cbd905f93e1af"}
Sep 16 12:15:37.114093 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:37.114064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ms47b" event={"ID":"c40a86af-2c1d-45a6-a409-2a4bb1d34252","Type":"ContainerStarted","Data":"d661cd5d008e0a7529bc6db1c8aabe9d5bce0c6d38cebc716c0df011c4efdcba"}
Sep 16 12:15:39.125551 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:39.125506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft" event={"ID":"5e30e789-4ae6-4ad8-b067-8c49e92857d8","Type":"ContainerStarted","Data":"1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a"}
Sep 16 12:15:39.127302 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:39.127272    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ms47b" event={"ID":"c40a86af-2c1d-45a6-a409-2a4bb1d34252","Type":"ContainerStarted","Data":"63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6"}
Sep 16 12:15:39.139860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:39.139775    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft" podStartSLOduration=1.614064642 podStartE2EDuration="3.139756614s" totalImagesPullingTime="1.525691972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:39.137810516 +0000 UTC m=+735.273132487" watchObservedRunningTime="2026-09-16 12:15:39.139756614 +0000 UTC m=+735.275078587"
Sep 16 12:15:39.159162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:39.159113    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-ms47b" podStartSLOduration=1.717887511 podStartE2EDuration="3.15909903s" totalImagesPullingTime="1.441211519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:39.157428924 +0000 UTC m=+735.292750895" watchObservedRunningTime="2026-09-16 12:15:39.15909903 +0000 UTC m=+735.294421001"
Sep 16 12:15:39.180391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:39.180360    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jmxft"]
Sep 16 12:15:41.138053 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:41.137998    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft" podUID="5e30e789-4ae6-4ad8-b067-8c49e92857d8" containerName="authorino" containerID="cri-o://1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a" gracePeriod=30
Sep 16 12:15:41.380839 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:41.380817    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft"
Sep 16 12:15:41.418465 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:41.418411    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv4zv\" (UniqueName: \"kubernetes.io/projected/5e30e789-4ae6-4ad8-b067-8c49e92857d8-kube-api-access-vv4zv\") pod \"5e30e789-4ae6-4ad8-b067-8c49e92857d8\" (UID: \"5e30e789-4ae6-4ad8-b067-8c49e92857d8\") "
Sep 16 12:15:41.420506 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:41.420485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e30e789-4ae6-4ad8-b067-8c49e92857d8-kube-api-access-vv4zv" (OuterVolumeSpecName: "kube-api-access-vv4zv") pod "5e30e789-4ae6-4ad8-b067-8c49e92857d8" (UID: "5e30e789-4ae6-4ad8-b067-8c49e92857d8"). InnerVolumeSpecName "kube-api-access-vv4zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:41.519130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:41.519105    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv4zv\" (UniqueName: \"kubernetes.io/projected/5e30e789-4ae6-4ad8-b067-8c49e92857d8-kube-api-access-vv4zv\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:15:42.144598 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.144561    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e30e789-4ae6-4ad8-b067-8c49e92857d8" containerID="1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a" exitCode=0
Sep 16 12:15:42.145025 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.144608    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft"
Sep 16 12:15:42.145025 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.144655    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft" event={"ID":"5e30e789-4ae6-4ad8-b067-8c49e92857d8","Type":"ContainerDied","Data":"1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a"}
Sep 16 12:15:42.145025 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.144699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jmxft" event={"ID":"5e30e789-4ae6-4ad8-b067-8c49e92857d8","Type":"ContainerDied","Data":"31d6e7ca8636d8d962c4ee23297eb8ee18f387ab5e8da7bf9d1cbd905f93e1af"}
Sep 16 12:15:42.145025 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.144719    2560 scope.go:117] "RemoveContainer" containerID="1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a"
Sep 16 12:15:42.154649 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.154629    2560 scope.go:117] "RemoveContainer" containerID="1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a"
Sep 16 12:15:42.155028 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:15:42.155005    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a\": container with ID starting with 1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a not found: ID does not exist" containerID="1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a"
Sep 16 12:15:42.155120 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.155036    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a"} err="failed to get container status \"1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a\": rpc error: code = NotFound desc = could not find container \"1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a\": container with ID starting with 1334645b4be4bac609a0f4411dc1e9b9af32c9b0b888a19cca7604b869228c3a not found: ID does not exist"
Sep 16 12:15:42.164550 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.164527    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jmxft"]
Sep 16 12:15:42.167750 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.167725    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jmxft"]
Sep 16 12:15:42.324625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:15:42.324598    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e30e789-4ae6-4ad8-b067-8c49e92857d8" path="/var/lib/kubelet/pods/5e30e789-4ae6-4ad8-b067-8c49e92857d8/volumes"
Sep 16 12:16:08.220233 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.220199    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zn6jq"]
Sep 16 12:16:08.220651 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.220535    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e30e789-4ae6-4ad8-b067-8c49e92857d8" containerName="authorino"
Sep 16 12:16:08.220651 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.220561    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e30e789-4ae6-4ad8-b067-8c49e92857d8" containerName="authorino"
Sep 16 12:16:08.220651 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.220624    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e30e789-4ae6-4ad8-b067-8c49e92857d8" containerName="authorino"
Sep 16 12:16:08.223707 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.223686    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zn6jq"
Sep 16 12:16:08.230631 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.230292    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zn6jq"]
Sep 16 12:16:08.316700 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.316669    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zn6jq"]
Sep 16 12:16:08.316967 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:16:08.316945    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4vnw6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-zn6jq" podUID="cdb354af-64c4-4f73-b685-bce78a37068e"
Sep 16 12:16:08.318445 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.318420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnw6\" (UniqueName: \"kubernetes.io/projected/cdb354af-64c4-4f73-b685-bce78a37068e-kube-api-access-4vnw6\") pod \"authorino-557f876f69-zn6jq\" (UID: \"cdb354af-64c4-4f73-b685-bce78a37068e\") " pod="kuadrant-system/authorino-557f876f69-zn6jq"
Sep 16 12:16:08.344671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.344643    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4h4ws"]
Sep 16 12:16:08.347921 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.347904    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:08.350393 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.350373    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 12:16:08.360711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.360691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4h4ws"]
Sep 16 12:16:08.418979 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.418947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnw6\" (UniqueName: \"kubernetes.io/projected/cdb354af-64c4-4f73-b685-bce78a37068e-kube-api-access-4vnw6\") pod \"authorino-557f876f69-zn6jq\" (UID: \"cdb354af-64c4-4f73-b685-bce78a37068e\") " pod="kuadrant-system/authorino-557f876f69-zn6jq"
Sep 16 12:16:08.419155 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.419001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/192c0cd5-3e75-4f2c-bee0-b432321f4656-tls-cert\") pod \"authorino-6d4dfc649f-4h4ws\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") " pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:08.419155 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.419095    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6b8b\" (UniqueName: \"kubernetes.io/projected/192c0cd5-3e75-4f2c-bee0-b432321f4656-kube-api-access-h6b8b\") pod \"authorino-6d4dfc649f-4h4ws\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") " pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:08.426790 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.426765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnw6\" (UniqueName: \"kubernetes.io/projected/cdb354af-64c4-4f73-b685-bce78a37068e-kube-api-access-4vnw6\") pod \"authorino-557f876f69-zn6jq\" (UID: \"cdb354af-64c4-4f73-b685-bce78a37068e\") " pod="kuadrant-system/authorino-557f876f69-zn6jq"
Sep 16 12:16:08.467398 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.467374    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4h4ws"]
Sep 16 12:16:08.467579 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:16:08.467563    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-h6b8b tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-4h4ws" podUID="192c0cd5-3e75-4f2c-bee0-b432321f4656"
Sep 16 12:16:08.494920 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.494870    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8589c4f846-5lthm"]
Sep 16 12:16:08.498252 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.498232    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:08.506919 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.506900    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8589c4f846-5lthm"]
Sep 16 12:16:08.520086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.520066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6b8b\" (UniqueName: \"kubernetes.io/projected/192c0cd5-3e75-4f2c-bee0-b432321f4656-kube-api-access-h6b8b\") pod \"authorino-6d4dfc649f-4h4ws\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") " pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:08.520171 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.520119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/192c0cd5-3e75-4f2c-bee0-b432321f4656-tls-cert\") pod \"authorino-6d4dfc649f-4h4ws\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") " pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:08.522577 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.522558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/192c0cd5-3e75-4f2c-bee0-b432321f4656-tls-cert\") pod \"authorino-6d4dfc649f-4h4ws\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") " pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:08.527662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.527641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6b8b\" (UniqueName: \"kubernetes.io/projected/192c0cd5-3e75-4f2c-bee0-b432321f4656-kube-api-access-h6b8b\") pod \"authorino-6d4dfc649f-4h4ws\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") " pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:08.621050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.621025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b97d2a-e12f-458c-9e26-175e64281815-tls-cert\") pod \"authorino-8589c4f846-5lthm\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") " pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:08.621136 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.621060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6764g\" (UniqueName: \"kubernetes.io/projected/96b97d2a-e12f-458c-9e26-175e64281815-kube-api-access-6764g\") pod \"authorino-8589c4f846-5lthm\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") " pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:08.722294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.722268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b97d2a-e12f-458c-9e26-175e64281815-tls-cert\") pod \"authorino-8589c4f846-5lthm\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") " pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:08.722397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.722304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6764g\" (UniqueName: \"kubernetes.io/projected/96b97d2a-e12f-458c-9e26-175e64281815-kube-api-access-6764g\") pod \"authorino-8589c4f846-5lthm\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") " pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:08.724856 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.724828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b97d2a-e12f-458c-9e26-175e64281815-tls-cert\") pod \"authorino-8589c4f846-5lthm\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") " pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:08.732600 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.732579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6764g\" (UniqueName: \"kubernetes.io/projected/96b97d2a-e12f-458c-9e26-175e64281815-kube-api-access-6764g\") pod \"authorino-8589c4f846-5lthm\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") " pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:08.807582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:08.807532    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:16:09.077027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.076955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8589c4f846-5lthm"]
Sep 16 12:16:09.080367 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:16:09.080339    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b97d2a_e12f_458c_9e26_175e64281815.slice/crio-f7b71b94da6003224a7327eb6e8822bddf008c6b60319e25f75427d97850c474 WatchSource:0}: Error finding container f7b71b94da6003224a7327eb6e8822bddf008c6b60319e25f75427d97850c474: Status 404 returned error can't find the container with id f7b71b94da6003224a7327eb6e8822bddf008c6b60319e25f75427d97850c474
Sep 16 12:16:09.271377 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.271353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zn6jq"
Sep 16 12:16:09.271377 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.271363    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8589c4f846-5lthm" event={"ID":"96b97d2a-e12f-458c-9e26-175e64281815","Type":"ContainerStarted","Data":"f7b71b94da6003224a7327eb6e8822bddf008c6b60319e25f75427d97850c474"}
Sep 16 12:16:09.271764 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.271542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:09.280499 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.280482    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zn6jq"
Sep 16 12:16:09.283652 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.283633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:09.429998 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.429970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/192c0cd5-3e75-4f2c-bee0-b432321f4656-tls-cert\") pod \"192c0cd5-3e75-4f2c-bee0-b432321f4656\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") "
Sep 16 12:16:09.430139 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.430005    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vnw6\" (UniqueName: \"kubernetes.io/projected/cdb354af-64c4-4f73-b685-bce78a37068e-kube-api-access-4vnw6\") pod \"cdb354af-64c4-4f73-b685-bce78a37068e\" (UID: \"cdb354af-64c4-4f73-b685-bce78a37068e\") "
Sep 16 12:16:09.430139 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.430083    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6b8b\" (UniqueName: \"kubernetes.io/projected/192c0cd5-3e75-4f2c-bee0-b432321f4656-kube-api-access-h6b8b\") pod \"192c0cd5-3e75-4f2c-bee0-b432321f4656\" (UID: \"192c0cd5-3e75-4f2c-bee0-b432321f4656\") "
Sep 16 12:16:09.433605 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.433572    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192c0cd5-3e75-4f2c-bee0-b432321f4656-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "192c0cd5-3e75-4f2c-bee0-b432321f4656" (UID: "192c0cd5-3e75-4f2c-bee0-b432321f4656"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:16:09.438014 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.437981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb354af-64c4-4f73-b685-bce78a37068e-kube-api-access-4vnw6" (OuterVolumeSpecName: "kube-api-access-4vnw6") pod "cdb354af-64c4-4f73-b685-bce78a37068e" (UID: "cdb354af-64c4-4f73-b685-bce78a37068e"). InnerVolumeSpecName "kube-api-access-4vnw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:09.439087 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.439052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192c0cd5-3e75-4f2c-bee0-b432321f4656-kube-api-access-h6b8b" (OuterVolumeSpecName: "kube-api-access-h6b8b") pod "192c0cd5-3e75-4f2c-bee0-b432321f4656" (UID: "192c0cd5-3e75-4f2c-bee0-b432321f4656"). InnerVolumeSpecName "kube-api-access-h6b8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:09.539313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.539280    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6b8b\" (UniqueName: \"kubernetes.io/projected/192c0cd5-3e75-4f2c-bee0-b432321f4656-kube-api-access-h6b8b\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:16:09.539313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.539308    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/192c0cd5-3e75-4f2c-bee0-b432321f4656-tls-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:16:09.539313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:09.539320    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vnw6\" (UniqueName: \"kubernetes.io/projected/cdb354af-64c4-4f73-b685-bce78a37068e-kube-api-access-4vnw6\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:16:10.278451 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.278415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8589c4f846-5lthm" event={"ID":"96b97d2a-e12f-458c-9e26-175e64281815","Type":"ContainerStarted","Data":"966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f"}
Sep 16 12:16:10.278875 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.278459    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4h4ws"
Sep 16 12:16:10.278875 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.278459    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zn6jq"
Sep 16 12:16:10.307431 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.307390    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8589c4f846-5lthm" podStartSLOduration=1.788111317 podStartE2EDuration="2.307376981s" totalImagesPullingTime="519.265664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:10.306086679 +0000 UTC m=+766.441408643" watchObservedRunningTime="2026-09-16 12:16:10.307376981 +0000 UTC m=+766.442698951"
Sep 16 12:16:10.344312 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.344283    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4h4ws"]
Sep 16 12:16:10.347777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.347735    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ms47b"]
Sep 16 12:16:10.347987 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.347951    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-ms47b" podUID="c40a86af-2c1d-45a6-a409-2a4bb1d34252" containerName="authorino" containerID="cri-o://63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6" gracePeriod=30
Sep 16 12:16:10.350276 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.350251    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4h4ws"]
Sep 16 12:16:10.368397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.368371    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zn6jq"]
Sep 16 12:16:10.370829 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.370805    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zn6jq"]
Sep 16 12:16:10.590773 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.590751    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ms47b"
Sep 16 12:16:10.750371 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.750345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkshh\" (UniqueName: \"kubernetes.io/projected/c40a86af-2c1d-45a6-a409-2a4bb1d34252-kube-api-access-lkshh\") pod \"c40a86af-2c1d-45a6-a409-2a4bb1d34252\" (UID: \"c40a86af-2c1d-45a6-a409-2a4bb1d34252\") "
Sep 16 12:16:10.752533 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.752499    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40a86af-2c1d-45a6-a409-2a4bb1d34252-kube-api-access-lkshh" (OuterVolumeSpecName: "kube-api-access-lkshh") pod "c40a86af-2c1d-45a6-a409-2a4bb1d34252" (UID: "c40a86af-2c1d-45a6-a409-2a4bb1d34252"). InnerVolumeSpecName "kube-api-access-lkshh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:10.850994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:10.850972    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkshh\" (UniqueName: \"kubernetes.io/projected/c40a86af-2c1d-45a6-a409-2a4bb1d34252-kube-api-access-lkshh\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:16:11.284599 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.284506    2560 generic.go:356] "Generic (PLEG): container finished" podID="c40a86af-2c1d-45a6-a409-2a4bb1d34252" containerID="63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6" exitCode=0
Sep 16 12:16:11.284599 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.284542    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ms47b"
Sep 16 12:16:11.284599 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.284588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ms47b" event={"ID":"c40a86af-2c1d-45a6-a409-2a4bb1d34252","Type":"ContainerDied","Data":"63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6"}
Sep 16 12:16:11.285192 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.284623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ms47b" event={"ID":"c40a86af-2c1d-45a6-a409-2a4bb1d34252","Type":"ContainerDied","Data":"d661cd5d008e0a7529bc6db1c8aabe9d5bce0c6d38cebc716c0df011c4efdcba"}
Sep 16 12:16:11.285192 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.284640    2560 scope.go:117] "RemoveContainer" containerID="63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6"
Sep 16 12:16:11.293548 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.293530    2560 scope.go:117] "RemoveContainer" containerID="63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6"
Sep 16 12:16:11.293811 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:16:11.293774    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6\": container with ID starting with 63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6 not found: ID does not exist" containerID="63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6"
Sep 16 12:16:11.293898 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.293815    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6"} err="failed to get container status \"63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6\": rpc error: code = NotFound desc = could not find container \"63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6\": container with ID starting with 63f086f8d3313f41a1c39cefd44dcd095179c42fd0da02974b6c315c69213cc6 not found: ID does not exist"
Sep 16 12:16:11.304077 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.304055    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ms47b"]
Sep 16 12:16:11.307023 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:11.307003    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ms47b"]
Sep 16 12:16:12.324365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:12.324335    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192c0cd5-3e75-4f2c-bee0-b432321f4656" path="/var/lib/kubelet/pods/192c0cd5-3e75-4f2c-bee0-b432321f4656/volumes"
Sep 16 12:16:12.324725 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:12.324542    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40a86af-2c1d-45a6-a409-2a4bb1d34252" path="/var/lib/kubelet/pods/c40a86af-2c1d-45a6-a409-2a4bb1d34252/volumes"
Sep 16 12:16:12.324831 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:12.324818    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb354af-64c4-4f73-b685-bce78a37068e" path="/var/lib/kubelet/pods/cdb354af-64c4-4f73-b685-bce78a37068e/volumes"
Sep 16 12:16:52.335510 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.335435    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"]
Sep 16 12:16:52.336021 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.336000    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40a86af-2c1d-45a6-a409-2a4bb1d34252" containerName="authorino"
Sep 16 12:16:52.336065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.336028    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40a86af-2c1d-45a6-a409-2a4bb1d34252" containerName="authorino"
Sep 16 12:16:52.336157 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.336145    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40a86af-2c1d-45a6-a409-2a4bb1d34252" containerName="authorino"
Sep 16 12:16:52.340100 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.340003    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.342745 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.342720    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-655rc\""
Sep 16 12:16:52.342912 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.342815    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 12:16:52.353414 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.353258    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"]
Sep 16 12:16:52.382957 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.382924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9b6\" (UniqueName: \"kubernetes.io/projected/8120361a-3d74-47be-b9a8-5ece86158766-kube-api-access-mt9b6\") pod \"payload-processing-5b54b7fbb7-8hfz8\" (UID: \"8120361a-3d74-47be-b9a8-5ece86158766\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.383131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.383060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8120361a-3d74-47be-b9a8-5ece86158766-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8hfz8\" (UID: \"8120361a-3d74-47be-b9a8-5ece86158766\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.483623 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.483586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9b6\" (UniqueName: \"kubernetes.io/projected/8120361a-3d74-47be-b9a8-5ece86158766-kube-api-access-mt9b6\") pod \"payload-processing-5b54b7fbb7-8hfz8\" (UID: \"8120361a-3d74-47be-b9a8-5ece86158766\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.483851 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.483701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8120361a-3d74-47be-b9a8-5ece86158766-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8hfz8\" (UID: \"8120361a-3d74-47be-b9a8-5ece86158766\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.484649 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.484624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8120361a-3d74-47be-b9a8-5ece86158766-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8hfz8\" (UID: \"8120361a-3d74-47be-b9a8-5ece86158766\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.491480 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.491450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9b6\" (UniqueName: \"kubernetes.io/projected/8120361a-3d74-47be-b9a8-5ece86158766-kube-api-access-mt9b6\") pod \"payload-processing-5b54b7fbb7-8hfz8\" (UID: \"8120361a-3d74-47be-b9a8-5ece86158766\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.655613 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.655577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:52.927076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:52.927051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"]
Sep 16 12:16:52.929658 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:16:52.929627    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8120361a_3d74_47be_b9a8_5ece86158766.slice/crio-196699c4400cc579d58c5cf2e2fb988a5081d2d474b614e4be7c66dcc5209323 WatchSource:0}: Error finding container 196699c4400cc579d58c5cf2e2fb988a5081d2d474b614e4be7c66dcc5209323: Status 404 returned error can't find the container with id 196699c4400cc579d58c5cf2e2fb988a5081d2d474b614e4be7c66dcc5209323
Sep 16 12:16:53.472042 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:53.471991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8" event={"ID":"8120361a-3d74-47be-b9a8-5ece86158766","Type":"ContainerStarted","Data":"196699c4400cc579d58c5cf2e2fb988a5081d2d474b614e4be7c66dcc5209323"}
Sep 16 12:16:55.483809 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:55.483719    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8" event={"ID":"8120361a-3d74-47be-b9a8-5ece86158766","Type":"ContainerStarted","Data":"da910ef934dfb615d7b39b5184bf74e5420ce456b598c913a09f19fc6a1168bb"}
Sep 16 12:16:55.484216 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:55.483829    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:16:55.500291 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:16:55.500234    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8" podStartSLOduration=1.337324659 podStartE2EDuration="3.50021815s" totalImagesPullingTime="2.162893491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:55.499808855 +0000 UTC m=+811.635130821" watchObservedRunningTime="2026-09-16 12:16:55.50021815 +0000 UTC m=+811.635540123"
Sep 16 12:17:06.489747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:06.489719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-8hfz8"
Sep 16 12:17:35.334332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.334295    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"]
Sep 16 12:17:35.342556 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.342531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.345128 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.345093    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:17:35.346213 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.346172    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:17:35.346348 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.346212    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c49mq\""
Sep 16 12:17:35.346348 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.346260    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 16 12:17:35.348535 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.348506    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"]
Sep 16 12:17:35.514275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.514246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.514438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.514285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxf4m\" (UniqueName: \"kubernetes.io/projected/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-kube-api-access-zxf4m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.514438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.514318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.514438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.514345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.514438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.514370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.514438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.514403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615233 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxf4m\" (UniqueName: \"kubernetes.io/projected/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-kube-api-access-zxf4m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615816 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615942 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.615942 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.615877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.617933 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.617913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.618349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.618328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.623318 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.623300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxf4m\" (UniqueName: \"kubernetes.io/projected/a023d8a6-4d8b-4c46-aa4b-40b876a5ee62-kube-api-access-zxf4m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-p4dsx\" (UID: \"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.656362 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.656338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:35.931085 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:35.929857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"]
Sep 16 12:17:35.935710 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:17:35.935337    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda023d8a6_4d8b_4c46_aa4b_40b876a5ee62.slice/crio-d11e4063ddc6b6f99f5e12bdfd3e1b43039b6feba1e2c1e6274a0fe1cdc7293d WatchSource:0}: Error finding container d11e4063ddc6b6f99f5e12bdfd3e1b43039b6feba1e2c1e6274a0fe1cdc7293d: Status 404 returned error can't find the container with id d11e4063ddc6b6f99f5e12bdfd3e1b43039b6feba1e2c1e6274a0fe1cdc7293d
Sep 16 12:17:36.671005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:36.670964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx" event={"ID":"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62","Type":"ContainerStarted","Data":"d11e4063ddc6b6f99f5e12bdfd3e1b43039b6feba1e2c1e6274a0fe1cdc7293d"}
Sep 16 12:17:37.677037 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:37.677002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx" event={"ID":"a023d8a6-4d8b-4c46-aa4b-40b876a5ee62","Type":"ContainerStarted","Data":"1209d4ab06e73a3616d920e2e125c69c00e72d4fd505249bd876a35dfca11aea"}
Sep 16 12:17:37.696120 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:37.696068    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx" podStartSLOduration=1.552935459 podStartE2EDuration="2.69605247s" totalImagesPullingTime="1.143117011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:37.692934436 +0000 UTC m=+853.828256406" watchObservedRunningTime="2026-09-16 12:17:37.69605247 +0000 UTC m=+853.831374442"
Sep 16 12:17:45.657050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:45.657015    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:45.657406 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:45.657069    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:45.669352 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:45.669324    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:45.733392 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:45.733369    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-p4dsx"
Sep 16 12:17:58.636043 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.636004    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"]
Sep 16 12:17:58.639493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.639469    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.641995 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.641968    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 12:17:58.649075 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.649044    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"]
Sep 16 12:17:58.693557 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.693534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.693669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.693563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094939c-9778-49ec-b2d7-e72f397086b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.693669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.693581    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.693669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.693606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.693669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.693655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vgw\" (UniqueName: \"kubernetes.io/projected/8094939c-9778-49ec-b2d7-e72f397086b9-kube-api-access-26vgw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.693867 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.693690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794063 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794235 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794235 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vgw\" (UniqueName: \"kubernetes.io/projected/8094939c-9778-49ec-b2d7-e72f397086b9-kube-api-access-26vgw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794235 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794235 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794235 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094939c-9778-49ec-b2d7-e72f397086b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794528 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794528 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.794678 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.794548    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.796510 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.796490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094939c-9778-49ec-b2d7-e72f397086b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.796777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.796761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094939c-9778-49ec-b2d7-e72f397086b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.802056 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.802037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vgw\" (UniqueName: \"kubernetes.io/projected/8094939c-9778-49ec-b2d7-e72f397086b9-kube-api-access-26vgw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nt8x\" (UID: \"8094939c-9778-49ec-b2d7-e72f397086b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:58.950177 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:58.950122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:17:59.223886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:59.223845    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"]
Sep 16 12:17:59.226587 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:17:59.226561    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8094939c_9778_49ec_b2d7_e72f397086b9.slice/crio-2b9ae48b57a71a94ae93223e70460ca6dd64c915c8b9c893211e216ae4fcf3a5 WatchSource:0}: Error finding container 2b9ae48b57a71a94ae93223e70460ca6dd64c915c8b9c893211e216ae4fcf3a5: Status 404 returned error can't find the container with id 2b9ae48b57a71a94ae93223e70460ca6dd64c915c8b9c893211e216ae4fcf3a5
Sep 16 12:17:59.789926 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:59.789849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x" event={"ID":"8094939c-9778-49ec-b2d7-e72f397086b9","Type":"ContainerStarted","Data":"1732b7734f9e1deb16b1e790af3d2c45e6baddfbc3859a0ad48805210e1cd2de"}
Sep 16 12:17:59.789926 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:59.789895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x" event={"ID":"8094939c-9778-49ec-b2d7-e72f397086b9","Type":"ContainerStarted","Data":"2b9ae48b57a71a94ae93223e70460ca6dd64c915c8b9c893211e216ae4fcf3a5"}
Sep 16 12:17:59.806701 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:17:59.806657    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x" podStartSLOduration=1.567269391 podStartE2EDuration="1.806642253s" totalImagesPullingTime="239.372862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:59.805540176 +0000 UTC m=+875.940862148" watchObservedRunningTime="2026-09-16 12:17:59.806642253 +0000 UTC m=+875.941964224"
Sep 16 12:18:08.830650 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.830618    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"]
Sep 16 12:18:08.836027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.836002    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.838647 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.838626    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 12:18:08.846569 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.846540    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"]
Sep 16 12:18:08.865633 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.865609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.865742 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.865645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.865742 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.865679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.865742 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.865702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvq2j\" (UniqueName: \"kubernetes.io/projected/20529458-de4e-41e5-bd25-82e02555b099-kube-api-access-qvq2j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.865742 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.865735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20529458-de4e-41e5-bd25-82e02555b099-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.865910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.865753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.950647 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.950621    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:18:08.950759 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.950694    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:18:08.962545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.962526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:18:08.966152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966250 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966250 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966366 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvq2j\" (UniqueName: \"kubernetes.io/projected/20529458-de4e-41e5-bd25-82e02555b099-kube-api-access-qvq2j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966366 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20529458-de4e-41e5-bd25-82e02555b099-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966366 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966595 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966595 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966595    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.966780 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.966759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.968563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.968543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20529458-de4e-41e5-bd25-82e02555b099-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.969019 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.968997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20529458-de4e-41e5-bd25-82e02555b099-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:08.973431 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:08.973398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvq2j\" (UniqueName: \"kubernetes.io/projected/20529458-de4e-41e5-bd25-82e02555b099-kube-api-access-qvq2j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9\" (UID: \"20529458-de4e-41e5-bd25-82e02555b099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:09.147653 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:09.147624    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:09.417157 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:09.417135    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"]
Sep 16 12:18:09.419873 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:18:09.419842    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20529458_de4e_41e5_bd25_82e02555b099.slice/crio-877cd0d5e4b8681c9f6595bb6a90aa5ea52e66f73d3438c2a7013b561c1c680c WatchSource:0}: Error finding container 877cd0d5e4b8681c9f6595bb6a90aa5ea52e66f73d3438c2a7013b561c1c680c: Status 404 returned error can't find the container with id 877cd0d5e4b8681c9f6595bb6a90aa5ea52e66f73d3438c2a7013b561c1c680c
Sep 16 12:18:09.838562 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:09.838528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9" event={"ID":"20529458-de4e-41e5-bd25-82e02555b099","Type":"ContainerStarted","Data":"c42102d4dede90b17a9f31193fbfd6ba961ee156f85af65df99646c7a8985bfb"}
Sep 16 12:18:09.839000 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:09.838585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9" event={"ID":"20529458-de4e-41e5-bd25-82e02555b099","Type":"ContainerStarted","Data":"877cd0d5e4b8681c9f6595bb6a90aa5ea52e66f73d3438c2a7013b561c1c680c"}
Sep 16 12:18:09.849861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:09.849839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nt8x"
Sep 16 12:18:09.857493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:09.857451    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9" podStartSLOduration=1.657794205 podStartE2EDuration="1.85743641s" totalImagesPullingTime="199.642205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:09.855614105 +0000 UTC m=+885.990936089" watchObservedRunningTime="2026-09-16 12:18:09.85743641 +0000 UTC m=+885.992758380"
Sep 16 12:18:19.148349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:19.148263    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:19.148349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:19.148323    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:19.161303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:19.161270    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:19.894888 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:19.894855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9"
Sep 16 12:18:23.731589 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.731552    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"]
Sep 16 12:18:23.735452 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.735429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.737906 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.737882    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 12:18:23.744966 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.744941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"]
Sep 16 12:18:23.794677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.794634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.794677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.794672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.794963 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.794813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cdm\" (UniqueName: \"kubernetes.io/projected/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-kube-api-access-p7cdm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.794963 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.794851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.794963 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.794900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.794963 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.794927    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896183 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cdm\" (UniqueName: \"kubernetes.io/projected/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-kube-api-access-p7cdm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896183 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896424 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896424 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896237    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896424 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896424 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896657 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.896746 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.896690    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.898666 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.898641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.899062 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.899041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:23.903675 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:23.903649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cdm\" (UniqueName: \"kubernetes.io/projected/7ceb40ee-1e7a-427f-8276-ff035f7f89c2-kube-api-access-p7cdm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s\" (UID: \"7ceb40ee-1e7a-427f-8276-ff035f7f89c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:24.046078 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:24.045989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:24.308249 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:24.308177    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:18:24.309247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:24.309226    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:18:24.323185 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:18:24.322858    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ceb40ee_1e7a_427f_8276_ff035f7f89c2.slice/crio-b9b03cbbe68836c0ed7fcfb42ecc28839a9d0cd4de377a92beb4fcfec0afe6e5 WatchSource:0}: Error finding container b9b03cbbe68836c0ed7fcfb42ecc28839a9d0cd4de377a92beb4fcfec0afe6e5: Status 404 returned error can't find the container with id b9b03cbbe68836c0ed7fcfb42ecc28839a9d0cd4de377a92beb4fcfec0afe6e5
Sep 16 12:18:24.325332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:24.325310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"]
Sep 16 12:18:24.910263 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:24.910231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s" event={"ID":"7ceb40ee-1e7a-427f-8276-ff035f7f89c2","Type":"ContainerStarted","Data":"ca368df73ffacac6dd2cf9c6971ef14d7f03c94b7ff5678939981b0fc8118344"}
Sep 16 12:18:24.910607 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:24.910270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s" event={"ID":"7ceb40ee-1e7a-427f-8276-ff035f7f89c2","Type":"ContainerStarted","Data":"b9b03cbbe68836c0ed7fcfb42ecc28839a9d0cd4de377a92beb4fcfec0afe6e5"}
Sep 16 12:18:24.926850 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:24.926774    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s" podStartSLOduration=1.740046551 podStartE2EDuration="1.92675622s" totalImagesPullingTime="186.709669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:24.925915829 +0000 UTC m=+901.061237811" watchObservedRunningTime="2026-09-16 12:18:24.92675622 +0000 UTC m=+901.062078192"
Sep 16 12:18:34.046311 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:34.046270    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:34.046812 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:34.046348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:34.058367 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:34.058345    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:18:34.970825 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:18:34.970779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s"
Sep 16 12:19:38.143076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.143038    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cc94d66d7-qb2tn"]
Sep 16 12:19:38.146701 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.146684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.153769 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.153707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cc94d66d7-qb2tn"]
Sep 16 12:19:38.292821 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.292772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbkd\" (UniqueName: \"kubernetes.io/projected/1208d70d-10bc-4fe9-acca-e2034f527fac-kube-api-access-xpbkd\") pod \"authorino-cc94d66d7-qb2tn\" (UID: \"1208d70d-10bc-4fe9-acca-e2034f527fac\") " pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.292949 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.292847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1208d70d-10bc-4fe9-acca-e2034f527fac-tls-cert\") pod \"authorino-cc94d66d7-qb2tn\" (UID: \"1208d70d-10bc-4fe9-acca-e2034f527fac\") " pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.394262 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.394194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbkd\" (UniqueName: \"kubernetes.io/projected/1208d70d-10bc-4fe9-acca-e2034f527fac-kube-api-access-xpbkd\") pod \"authorino-cc94d66d7-qb2tn\" (UID: \"1208d70d-10bc-4fe9-acca-e2034f527fac\") " pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.394262 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.394242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1208d70d-10bc-4fe9-acca-e2034f527fac-tls-cert\") pod \"authorino-cc94d66d7-qb2tn\" (UID: \"1208d70d-10bc-4fe9-acca-e2034f527fac\") " pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.396821 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.396782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1208d70d-10bc-4fe9-acca-e2034f527fac-tls-cert\") pod \"authorino-cc94d66d7-qb2tn\" (UID: \"1208d70d-10bc-4fe9-acca-e2034f527fac\") " pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.402320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.402294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbkd\" (UniqueName: \"kubernetes.io/projected/1208d70d-10bc-4fe9-acca-e2034f527fac-kube-api-access-xpbkd\") pod \"authorino-cc94d66d7-qb2tn\" (UID: \"1208d70d-10bc-4fe9-acca-e2034f527fac\") " pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.456826 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.456805    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc94d66d7-qb2tn"
Sep 16 12:19:38.724115 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.724091    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cc94d66d7-qb2tn"]
Sep 16 12:19:38.726889 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:19:38.726863    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1208d70d_10bc_4fe9_acca_e2034f527fac.slice/crio-bab1447e1a09dbb472a33b4b1942b6abd7d8aacb352668616836f16d369c331d WatchSource:0}: Error finding container bab1447e1a09dbb472a33b4b1942b6abd7d8aacb352668616836f16d369c331d: Status 404 returned error can't find the container with id bab1447e1a09dbb472a33b4b1942b6abd7d8aacb352668616836f16d369c331d
Sep 16 12:19:38.728307 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:38.728286    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:19:39.265435 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:39.265402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc94d66d7-qb2tn" event={"ID":"1208d70d-10bc-4fe9-acca-e2034f527fac","Type":"ContainerStarted","Data":"bab1447e1a09dbb472a33b4b1942b6abd7d8aacb352668616836f16d369c331d"}
Sep 16 12:19:40.280540 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.280501    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc94d66d7-qb2tn" event={"ID":"1208d70d-10bc-4fe9-acca-e2034f527fac","Type":"ContainerStarted","Data":"3dfbb58578f13730efe9be061e454f117e855d5a9e45f8607a245987fcde7e1f"}
Sep 16 12:19:40.297384 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.297334    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cc94d66d7-qb2tn" podStartSLOduration=1.651125803 podStartE2EDuration="2.297319978s" totalImagesPullingTime="646.194175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:40.296503125 +0000 UTC m=+976.431825097" watchObservedRunningTime="2026-09-16 12:19:40.297319978 +0000 UTC m=+976.432641948"
Sep 16 12:19:40.326326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.326286    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8589c4f846-5lthm"]
Sep 16 12:19:40.326545 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.326513    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8589c4f846-5lthm" podUID="96b97d2a-e12f-458c-9e26-175e64281815" containerName="authorino" containerID="cri-o://966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f" gracePeriod=30
Sep 16 12:19:40.584354 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.584324    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:19:40.717255 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.717224    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6764g\" (UniqueName: \"kubernetes.io/projected/96b97d2a-e12f-458c-9e26-175e64281815-kube-api-access-6764g\") pod \"96b97d2a-e12f-458c-9e26-175e64281815\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") "
Sep 16 12:19:40.717423 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.717285    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b97d2a-e12f-458c-9e26-175e64281815-tls-cert\") pod \"96b97d2a-e12f-458c-9e26-175e64281815\" (UID: \"96b97d2a-e12f-458c-9e26-175e64281815\") "
Sep 16 12:19:40.719603 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.719562    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b97d2a-e12f-458c-9e26-175e64281815-kube-api-access-6764g" (OuterVolumeSpecName: "kube-api-access-6764g") pod "96b97d2a-e12f-458c-9e26-175e64281815" (UID: "96b97d2a-e12f-458c-9e26-175e64281815"). InnerVolumeSpecName "kube-api-access-6764g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:40.727942 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.727915    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b97d2a-e12f-458c-9e26-175e64281815-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "96b97d2a-e12f-458c-9e26-175e64281815" (UID: "96b97d2a-e12f-458c-9e26-175e64281815"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:40.819018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.818931    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6764g\" (UniqueName: \"kubernetes.io/projected/96b97d2a-e12f-458c-9e26-175e64281815-kube-api-access-6764g\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:19:40.819018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:40.818970    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b97d2a-e12f-458c-9e26-175e64281815-tls-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:19:41.287050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.287014    2560 generic.go:356] "Generic (PLEG): container finished" podID="96b97d2a-e12f-458c-9e26-175e64281815" containerID="966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f" exitCode=0
Sep 16 12:19:41.287470 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.287062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8589c4f846-5lthm"
Sep 16 12:19:41.287470 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.287092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8589c4f846-5lthm" event={"ID":"96b97d2a-e12f-458c-9e26-175e64281815","Type":"ContainerDied","Data":"966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f"}
Sep 16 12:19:41.287470 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.287135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8589c4f846-5lthm" event={"ID":"96b97d2a-e12f-458c-9e26-175e64281815","Type":"ContainerDied","Data":"f7b71b94da6003224a7327eb6e8822bddf008c6b60319e25f75427d97850c474"}
Sep 16 12:19:41.287470 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.287151    2560 scope.go:117] "RemoveContainer" containerID="966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f"
Sep 16 12:19:41.296240 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.296218    2560 scope.go:117] "RemoveContainer" containerID="966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f"
Sep 16 12:19:41.296536 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:19:41.296516    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f\": container with ID starting with 966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f not found: ID does not exist" containerID="966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f"
Sep 16 12:19:41.296612 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.296545    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f"} err="failed to get container status \"966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f\": rpc error: code = NotFound desc = could not find container \"966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f\": container with ID starting with 966f44448f140a6cc3ac22e7299f98fabf0e194eae7ad5b6f421e9f6d70a611f not found: ID does not exist"
Sep 16 12:19:41.310293 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.310269    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8589c4f846-5lthm"]
Sep 16 12:19:41.313811 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:41.313777    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8589c4f846-5lthm"]
Sep 16 12:19:42.324244 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:19:42.324211    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b97d2a-e12f-458c-9e26-175e64281815" path="/var/lib/kubelet/pods/96b97d2a-e12f-458c-9e26-175e64281815/volumes"
Sep 16 12:20:05.693850 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.693816    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"]
Sep 16 12:20:05.694247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.694236    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b97d2a-e12f-458c-9e26-175e64281815" containerName="authorino"
Sep 16 12:20:05.694291 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.694250    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b97d2a-e12f-458c-9e26-175e64281815" containerName="authorino"
Sep 16 12:20:05.694329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.694308    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b97d2a-e12f-458c-9e26-175e64281815" containerName="authorino"
Sep 16 12:20:05.697464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.697442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:05.699957 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.699936    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f79e22\""
Sep 16 12:20:05.700277 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.699978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f79e22-dockercfg-pmplg\""
Sep 16 12:20:05.706860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.706837    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"]
Sep 16 12:20:05.721558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.721532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a487f889-d060-4b25-8909-93325abee717-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:05.721697 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.721622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twp75\" (UniqueName: \"kubernetes.io/projected/a487f889-d060-4b25-8909-93325abee717-kube-api-access-twp75\") pod \"payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:05.822281 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.822245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twp75\" (UniqueName: \"kubernetes.io/projected/a487f889-d060-4b25-8909-93325abee717-kube-api-access-twp75\") pod \"payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:05.822441 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.822328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a487f889-d060-4b25-8909-93325abee717-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:05.823076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.823047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a487f889-d060-4b25-8909-93325abee717-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:05.830445 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:05.830424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twp75\" (UniqueName: \"kubernetes.io/projected/a487f889-d060-4b25-8909-93325abee717-kube-api-access-twp75\") pod \"payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:06.009038 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:06.008946    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:06.284975 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:06.284950    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"]
Sep 16 12:20:06.287559 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:06.287522    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda487f889_d060_4b25_8909_93325abee717.slice/crio-a73dd74bb6090f074804a568ba26a5127f1186ef1b012b0252ecfcac17f4c2f7 WatchSource:0}: Error finding container a73dd74bb6090f074804a568ba26a5127f1186ef1b012b0252ecfcac17f4c2f7: Status 404 returned error can't find the container with id a73dd74bb6090f074804a568ba26a5127f1186ef1b012b0252ecfcac17f4c2f7
Sep 16 12:20:06.408367 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:06.408327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf" event={"ID":"a487f889-d060-4b25-8909-93325abee717","Type":"ContainerStarted","Data":"83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5"}
Sep 16 12:20:06.408515 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:06.408392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf" event={"ID":"a487f889-d060-4b25-8909-93325abee717","Type":"ContainerStarted","Data":"a73dd74bb6090f074804a568ba26a5127f1186ef1b012b0252ecfcac17f4c2f7"}
Sep 16 12:20:06.408578 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:06.408516    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:06.423110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:06.423059    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf" podStartSLOduration=1.42304284 podStartE2EDuration="1.42304284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:06.422248269 +0000 UTC m=+1002.557570252" watchObservedRunningTime="2026-09-16 12:20:06.42304284 +0000 UTC m=+1002.558364810"
Sep 16 12:20:07.620115 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.620078    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"]
Sep 16 12:20:07.623757 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.623740    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.626312 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.626277    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-739ee7\""
Sep 16 12:20:07.626444 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.626387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-739ee7-dockercfg-287t8\""
Sep 16 12:20:07.634841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.634816    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"]
Sep 16 12:20:07.636938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.636919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9qn\" (UniqueName: \"kubernetes.io/projected/f98864b0-bcec-46b8-95f1-fba92b3234ed-kube-api-access-6t9qn\") pod \"payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.637033 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.636946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98864b0-bcec-46b8-95f1-fba92b3234ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.659568 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.659544    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"]
Sep 16 12:20:07.662841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.662826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:07.673032 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.673009    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"]
Sep 16 12:20:07.737702 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.737659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9qn\" (UniqueName: \"kubernetes.io/projected/f98864b0-bcec-46b8-95f1-fba92b3234ed-kube-api-access-6t9qn\") pod \"payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.737862 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.737718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98864b0-bcec-46b8-95f1-fba92b3234ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.737862 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.737763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ed33c5-b4fa-4bc5-9357-4a12399784e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:07.737862 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.737831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgbp\" (UniqueName: \"kubernetes.io/projected/22ed33c5-b4fa-4bc5-9357-4a12399784e1-kube-api-access-7wgbp\") pod \"payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:07.738523 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.738503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98864b0-bcec-46b8-95f1-fba92b3234ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.746136 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.746100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9qn\" (UniqueName: \"kubernetes.io/projected/f98864b0-bcec-46b8-95f1-fba92b3234ed-kube-api-access-6t9qn\") pod \"payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.838579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.838546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ed33c5-b4fa-4bc5-9357-4a12399784e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:07.838723 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.838595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgbp\" (UniqueName: \"kubernetes.io/projected/22ed33c5-b4fa-4bc5-9357-4a12399784e1-kube-api-access-7wgbp\") pod \"payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:07.839406 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.839386    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ed33c5-b4fa-4bc5-9357-4a12399784e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:07.847095 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.847073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgbp\" (UniqueName: \"kubernetes.io/projected/22ed33c5-b4fa-4bc5-9357-4a12399784e1-kube-api-access-7wgbp\") pod \"payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:07.935543 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.935458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:07.973007 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:07.972971    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:08.242879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.242851    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"]
Sep 16 12:20:08.245012 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:08.244981    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98864b0_bcec_46b8_95f1_fba92b3234ed.slice/crio-0d77b6d142d621a1282022cc6083478653a1c1adae29a4ea57e772bdda6daffa WatchSource:0}: Error finding container 0d77b6d142d621a1282022cc6083478653a1c1adae29a4ea57e772bdda6daffa: Status 404 returned error can't find the container with id 0d77b6d142d621a1282022cc6083478653a1c1adae29a4ea57e772bdda6daffa
Sep 16 12:20:08.284097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.284073    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"]
Sep 16 12:20:08.285545 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:08.285521    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ed33c5_b4fa_4bc5_9357_4a12399784e1.slice/crio-50f348f28a31e098dbef87f4c6ba86264ed834b64d1f7dacd134b7ba37793c6d WatchSource:0}: Error finding container 50f348f28a31e098dbef87f4c6ba86264ed834b64d1f7dacd134b7ba37793c6d: Status 404 returned error can't find the container with id 50f348f28a31e098dbef87f4c6ba86264ed834b64d1f7dacd134b7ba37793c6d
Sep 16 12:20:08.433879 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.433842    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v" event={"ID":"f98864b0-bcec-46b8-95f1-fba92b3234ed","Type":"ContainerStarted","Data":"d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769"}
Sep 16 12:20:08.434058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.433936    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v" event={"ID":"f98864b0-bcec-46b8-95f1-fba92b3234ed","Type":"ContainerStarted","Data":"0d77b6d142d621a1282022cc6083478653a1c1adae29a4ea57e772bdda6daffa"}
Sep 16 12:20:08.434058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.433968    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:08.435894 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.435862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248" event={"ID":"22ed33c5-b4fa-4bc5-9357-4a12399784e1","Type":"ContainerStarted","Data":"d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992"}
Sep 16 12:20:08.436027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.435910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248" event={"ID":"22ed33c5-b4fa-4bc5-9357-4a12399784e1","Type":"ContainerStarted","Data":"50f348f28a31e098dbef87f4c6ba86264ed834b64d1f7dacd134b7ba37793c6d"}
Sep 16 12:20:08.436027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.435956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:08.451523 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.451482    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v" podStartSLOduration=1.451468257 podStartE2EDuration="1.451468257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:08.449562052 +0000 UTC m=+1004.584884025" watchObservedRunningTime="2026-09-16 12:20:08.451468257 +0000 UTC m=+1004.586790255"
Sep 16 12:20:08.465289 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:08.465241    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248" podStartSLOduration=1.4652240220000001 podStartE2EDuration="1.465224022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:08.463869399 +0000 UTC m=+1004.599191371" watchObservedRunningTime="2026-09-16 12:20:08.465224022 +0000 UTC m=+1004.600545997"
Sep 16 12:20:10.210442 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.210401    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"]
Sep 16 12:20:10.214515 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.214492    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.217017 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.216997    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-68f7cd\""
Sep 16 12:20:10.217113 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.217054    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-68f7cd-dockercfg-j5f4j\""
Sep 16 12:20:10.224583 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.224544    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"]
Sep 16 12:20:10.227461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.227436    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"]
Sep 16 12:20:10.231409 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.231391    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.242001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.241747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"]
Sep 16 12:20:10.258900 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.258871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51be8bcf-f26b-4f65-901e-f4b16921995c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.259008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.258922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmmk\" (UniqueName: \"kubernetes.io/projected/51be8bcf-f26b-4f65-901e-f4b16921995c-kube-api-access-ttmmk\") pod \"payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.259074 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.259028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce24f8c7-fe43-45cd-add7-74ab52821417-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.259130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.259106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdvm\" (UniqueName: \"kubernetes.io/projected/ce24f8c7-fe43-45cd-add7-74ab52821417-kube-api-access-xhdvm\") pod \"payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.360632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.360593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdvm\" (UniqueName: \"kubernetes.io/projected/ce24f8c7-fe43-45cd-add7-74ab52821417-kube-api-access-xhdvm\") pod \"payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.360845 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.360656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51be8bcf-f26b-4f65-901e-f4b16921995c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.360845 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.360702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmmk\" (UniqueName: \"kubernetes.io/projected/51be8bcf-f26b-4f65-901e-f4b16921995c-kube-api-access-ttmmk\") pod \"payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.360981 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.360854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce24f8c7-fe43-45cd-add7-74ab52821417-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.361592 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.361572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce24f8c7-fe43-45cd-add7-74ab52821417-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.361646 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.361600    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51be8bcf-f26b-4f65-901e-f4b16921995c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.372067 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.372041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmmk\" (UniqueName: \"kubernetes.io/projected/51be8bcf-f26b-4f65-901e-f4b16921995c-kube-api-access-ttmmk\") pod \"payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.372406 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.372387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdvm\" (UniqueName: \"kubernetes.io/projected/ce24f8c7-fe43-45cd-add7-74ab52821417-kube-api-access-xhdvm\") pod \"payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.528305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.528224    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:10.544677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.544650    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:10.826254 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.826224    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"]
Sep 16 12:20:10.827858 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:10.827825    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51be8bcf_f26b_4f65_901e_f4b16921995c.slice/crio-d0e95caa001cc5313c298ebb2cf89df8f22579d8f7781b69a1e7e61d618b853e WatchSource:0}: Error finding container d0e95caa001cc5313c298ebb2cf89df8f22579d8f7781b69a1e7e61d618b853e: Status 404 returned error can't find the container with id d0e95caa001cc5313c298ebb2cf89df8f22579d8f7781b69a1e7e61d618b853e
Sep 16 12:20:10.863774 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:10.863751    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"]
Sep 16 12:20:10.865888 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:10.865861    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce24f8c7_fe43_45cd_add7_74ab52821417.slice/crio-e8667480b04353c50ff2e45bebc513d9d427c389d1b0c5aebd2618a89d851f9b WatchSource:0}: Error finding container e8667480b04353c50ff2e45bebc513d9d427c389d1b0c5aebd2618a89d851f9b: Status 404 returned error can't find the container with id e8667480b04353c50ff2e45bebc513d9d427c389d1b0c5aebd2618a89d851f9b
Sep 16 12:20:11.454094 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.454053    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q" event={"ID":"ce24f8c7-fe43-45cd-add7-74ab52821417","Type":"ContainerStarted","Data":"ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3"}
Sep 16 12:20:11.454579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.454122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q" event={"ID":"ce24f8c7-fe43-45cd-add7-74ab52821417","Type":"ContainerStarted","Data":"e8667480b04353c50ff2e45bebc513d9d427c389d1b0c5aebd2618a89d851f9b"}
Sep 16 12:20:11.454579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.454150    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:11.455849 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.455818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj" event={"ID":"51be8bcf-f26b-4f65-901e-f4b16921995c","Type":"ContainerStarted","Data":"d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd"}
Sep 16 12:20:11.455967 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.455860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj" event={"ID":"51be8bcf-f26b-4f65-901e-f4b16921995c","Type":"ContainerStarted","Data":"d0e95caa001cc5313c298ebb2cf89df8f22579d8f7781b69a1e7e61d618b853e"}
Sep 16 12:20:11.455967 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.455904    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:11.476735 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.476691    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q" podStartSLOduration=1.476676641 podStartE2EDuration="1.476676641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:11.474723493 +0000 UTC m=+1007.610045488" watchObservedRunningTime="2026-09-16 12:20:11.476676641 +0000 UTC m=+1007.611998612"
Sep 16 12:20:11.490086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:11.490006    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj" podStartSLOduration=1.489990752 podStartE2EDuration="1.489990752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:11.489863021 +0000 UTC m=+1007.625184993" watchObservedRunningTime="2026-09-16 12:20:11.489990752 +0000 UTC m=+1007.625312729"
Sep 16 12:20:15.723034 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.723001    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"]
Sep 16 12:20:15.727442 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.727426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:15.730050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.730030    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-00a141-dockercfg-gqspw\""
Sep 16 12:20:15.730146 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.730028    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-00a141\""
Sep 16 12:20:15.737598 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.737575    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"]
Sep 16 12:20:15.763005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.762982    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"]
Sep 16 12:20:15.766243 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.766226    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:15.775566 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.775542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"]
Sep 16 12:20:15.818313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.818287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d0f7950-9a2d-4834-abe6-95914cec587d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:15.818472 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.818337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2vt\" (UniqueName: \"kubernetes.io/projected/3d0f7950-9a2d-4834-abe6-95914cec587d-kube-api-access-7f2vt\") pod \"payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:15.818472 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.818409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dfz\" (UniqueName: \"kubernetes.io/projected/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-kube-api-access-x2dfz\") pod \"payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") " pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:15.818472 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.818468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-plugins-config-volume\") pod \"payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") " pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:15.918947 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.918909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d0f7950-9a2d-4834-abe6-95914cec587d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:15.919113 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.918958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2vt\" (UniqueName: \"kubernetes.io/projected/3d0f7950-9a2d-4834-abe6-95914cec587d-kube-api-access-7f2vt\") pod \"payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:15.919113 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.918988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dfz\" (UniqueName: \"kubernetes.io/projected/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-kube-api-access-x2dfz\") pod \"payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") " pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:15.919113 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.919032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-plugins-config-volume\") pod \"payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") " pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:15.919677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.919654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d0f7950-9a2d-4834-abe6-95914cec587d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:15.919996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.919867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-plugins-config-volume\") pod \"payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") " pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:15.927096 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.927048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dfz\" (UniqueName: \"kubernetes.io/projected/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-kube-api-access-x2dfz\") pod \"payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") " pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:15.927365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:15.927192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2vt\" (UniqueName: \"kubernetes.io/projected/3d0f7950-9a2d-4834-abe6-95914cec587d-kube-api-access-7f2vt\") pod \"payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:16.039463 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.039378    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:16.077627 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.077598    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:16.318704 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.318645    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"]
Sep 16 12:20:16.321326 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:16.321285    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0f7950_9a2d_4834_abe6_95914cec587d.slice/crio-d711984d5de3490d168d785490a0c3932f628626a81ff03a37fba8edc4483565 WatchSource:0}: Error finding container d711984d5de3490d168d785490a0c3932f628626a81ff03a37fba8edc4483565: Status 404 returned error can't find the container with id d711984d5de3490d168d785490a0c3932f628626a81ff03a37fba8edc4483565
Sep 16 12:20:16.366908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.366879    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"]
Sep 16 12:20:16.369836 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:16.369807    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb17a61a_9daa_44cb_ba18_fdce6a08c9ae.slice/crio-15e72c3c31a96d5f1343ac662365260db0a3c5b34195cb2eefcedce5430ac289 WatchSource:0}: Error finding container 15e72c3c31a96d5f1343ac662365260db0a3c5b34195cb2eefcedce5430ac289: Status 404 returned error can't find the container with id 15e72c3c31a96d5f1343ac662365260db0a3c5b34195cb2eefcedce5430ac289
Sep 16 12:20:16.483574 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.483536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr" event={"ID":"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae","Type":"ContainerStarted","Data":"2ddc49ce9ee19f31f586c4276f96647b121e10e7721bdfd8ecc01e32ff5c13b2"}
Sep 16 12:20:16.483756 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.483597    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr" event={"ID":"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae","Type":"ContainerStarted","Data":"15e72c3c31a96d5f1343ac662365260db0a3c5b34195cb2eefcedce5430ac289"}
Sep 16 12:20:16.483756 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.483629    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:16.485449 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.485417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j" event={"ID":"3d0f7950-9a2d-4834-abe6-95914cec587d","Type":"ContainerStarted","Data":"0473b2bd40587e5174dc60211d4067cec8a6dcce3bcec9876ac1535dc8f0a6e0"}
Sep 16 12:20:16.485563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.485467    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:16.485563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.485493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j" event={"ID":"3d0f7950-9a2d-4834-abe6-95914cec587d","Type":"ContainerStarted","Data":"d711984d5de3490d168d785490a0c3932f628626a81ff03a37fba8edc4483565"}
Sep 16 12:20:16.505742 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.505677    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr" podStartSLOduration=1.5056588469999999 podStartE2EDuration="1.505658847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:16.505293647 +0000 UTC m=+1012.640615619" watchObservedRunningTime="2026-09-16 12:20:16.505658847 +0000 UTC m=+1012.640980820"
Sep 16 12:20:16.526439 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.526380    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j" podStartSLOduration=1.526360795 podStartE2EDuration="1.526360795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:16.523861841 +0000 UTC m=+1012.659183819" watchObservedRunningTime="2026-09-16 12:20:16.526360795 +0000 UTC m=+1012.661682775"
Sep 16 12:20:16.616681 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.616652    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"]
Sep 16 12:20:16.620428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.620406    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:16.623225 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.623206    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5fb73413-dockercfg-w48zx\""
Sep 16 12:20:16.623355 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.623258    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5fb73413\""
Sep 16 12:20:16.632438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.632413    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"]
Sep 16 12:20:16.732402 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.732364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64ds\" (UniqueName: \"kubernetes.io/projected/1a3c5251-3464-4de5-a39b-348a55da48d8-kube-api-access-z64ds\") pod \"payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") " pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:16.732845 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.732537    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c5251-3464-4de5-a39b-348a55da48d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") " pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:16.833758 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.833710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c5251-3464-4de5-a39b-348a55da48d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") " pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:16.833947 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.833837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64ds\" (UniqueName: \"kubernetes.io/projected/1a3c5251-3464-4de5-a39b-348a55da48d8-kube-api-access-z64ds\") pod \"payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") " pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:16.834616 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.834587    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c5251-3464-4de5-a39b-348a55da48d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") " pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:16.844416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.844381    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64ds\" (UniqueName: \"kubernetes.io/projected/1a3c5251-3464-4de5-a39b-348a55da48d8-kube-api-access-z64ds\") pod \"payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") " pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:16.934085 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:16.934005    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:17.211913 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:17.211884    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"]
Sep 16 12:20:17.214136 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:17.214105    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3c5251_3464_4de5_a39b_348a55da48d8.slice/crio-927a669881869b72a11804c5eda679a010a88dcb11f9678f24b449634a8a1f78 WatchSource:0}: Error finding container 927a669881869b72a11804c5eda679a010a88dcb11f9678f24b449634a8a1f78: Status 404 returned error can't find the container with id 927a669881869b72a11804c5eda679a010a88dcb11f9678f24b449634a8a1f78
Sep 16 12:20:17.418895 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:17.418862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:20:17.494916 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:17.494830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d" event={"ID":"1a3c5251-3464-4de5-a39b-348a55da48d8","Type":"ContainerStarted","Data":"b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60"}
Sep 16 12:20:17.494916 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:17.494909    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d" event={"ID":"1a3c5251-3464-4de5-a39b-348a55da48d8","Type":"ContainerStarted","Data":"927a669881869b72a11804c5eda679a010a88dcb11f9678f24b449634a8a1f78"}
Sep 16 12:20:18.500628 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:18.500599    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:19.444704 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:19.444678    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:20:19.444913 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:19.444748    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:20:19.464302 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:19.464254    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d" podStartSLOduration=3.464240123 podStartE2EDuration="3.464240123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:17.514895314 +0000 UTC m=+1013.650217286" watchObservedRunningTime="2026-09-16 12:20:19.464240123 +0000 UTC m=+1015.599562093"
Sep 16 12:20:22.462639 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:22.462604    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:20:22.463040 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:22.462819    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:20:27.071620 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.071588    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"]
Sep 16 12:20:27.076326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.076308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.078972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.078945    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ef3317\""
Sep 16 12:20:27.078972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.078966    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ef3317-dockercfg-xpnfb\""
Sep 16 12:20:27.085315 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.085294    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"]
Sep 16 12:20:27.132781 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.132749    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"]
Sep 16 12:20:27.136437 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.136420    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.147153 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.147121    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"]
Sep 16 12:20:27.245973 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.245939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.246134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.245992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c928568-e9da-4add-bebb-43cf7bc5208e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.246134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.246085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzf2\" (UniqueName: \"kubernetes.io/projected/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-kube-api-access-bvzf2\") pod \"payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.246212 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.246193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvc22\" (UniqueName: \"kubernetes.io/projected/2c928568-e9da-4add-bebb-43cf7bc5208e-kube-api-access-nvc22\") pod \"payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.347558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.347528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvc22\" (UniqueName: \"kubernetes.io/projected/2c928568-e9da-4add-bebb-43cf7bc5208e-kube-api-access-nvc22\") pod \"payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.347709 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.347582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.347709 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.347627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c928568-e9da-4add-bebb-43cf7bc5208e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.347709 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.347679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzf2\" (UniqueName: \"kubernetes.io/projected/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-kube-api-access-bvzf2\") pod \"payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.348572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.348551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.348629 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.348551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c928568-e9da-4add-bebb-43cf7bc5208e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.356234 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.356206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvc22\" (UniqueName: \"kubernetes.io/projected/2c928568-e9da-4add-bebb-43cf7bc5208e-kube-api-access-nvc22\") pod \"payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.356538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.356518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzf2\" (UniqueName: \"kubernetes.io/projected/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-kube-api-access-bvzf2\") pod \"payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.388519 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.388458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:27.450278 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.450245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:27.496930 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.496821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:20:27.497280 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.497262    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:20:27.661934 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.661905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"]
Sep 16 12:20:27.662927 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:27.662895    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c21f44_34b8_4ac5_9cfa_f8aa73909b1c.slice/crio-4a04c1e7b0070e6705feb6963652d2592219a633be18ce1ba8b3219209b0cadb WatchSource:0}: Error finding container 4a04c1e7b0070e6705feb6963652d2592219a633be18ce1ba8b3219209b0cadb: Status 404 returned error can't find the container with id 4a04c1e7b0070e6705feb6963652d2592219a633be18ce1ba8b3219209b0cadb
Sep 16 12:20:27.733615 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:27.733585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"]
Sep 16 12:20:27.736461 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:27.736430    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c928568_e9da_4add_bebb_43cf7bc5208e.slice/crio-2c1e9341c2aac97dec8a60cf1ae41f9af17c76fe015d0dda162e93b7be965356 WatchSource:0}: Error finding container 2c1e9341c2aac97dec8a60cf1ae41f9af17c76fe015d0dda162e93b7be965356: Status 404 returned error can't find the container with id 2c1e9341c2aac97dec8a60cf1ae41f9af17c76fe015d0dda162e93b7be965356
Sep 16 12:20:28.566123 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.566087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4" event={"ID":"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c","Type":"ContainerStarted","Data":"fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8"}
Sep 16 12:20:28.566558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.566137    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4" event={"ID":"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c","Type":"ContainerStarted","Data":"4a04c1e7b0070e6705feb6963652d2592219a633be18ce1ba8b3219209b0cadb"}
Sep 16 12:20:28.566558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.566179    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:28.568002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.567973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw" event={"ID":"2c928568-e9da-4add-bebb-43cf7bc5208e","Type":"ContainerStarted","Data":"f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d"}
Sep 16 12:20:28.568116 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.568030    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:28.568116 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.568052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw" event={"ID":"2c928568-e9da-4add-bebb-43cf7bc5208e","Type":"ContainerStarted","Data":"2c1e9341c2aac97dec8a60cf1ae41f9af17c76fe015d0dda162e93b7be965356"}
Sep 16 12:20:28.585765 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.585723    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4" podStartSLOduration=1.5857092000000002 podStartE2EDuration="1.5857092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:28.583635326 +0000 UTC m=+1024.718957309" watchObservedRunningTime="2026-09-16 12:20:28.5857092 +0000 UTC m=+1024.721031168"
Sep 16 12:20:28.600957 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:28.600910    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw" podStartSLOduration=1.6008936390000001 podStartE2EDuration="1.600893639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:28.599143782 +0000 UTC m=+1024.734465754" watchObservedRunningTime="2026-09-16 12:20:28.600893639 +0000 UTC m=+1024.736215611"
Sep 16 12:20:29.511044 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:29.511017    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:39.575972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:39.575938    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:20:39.576372 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:39.576234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:20:43.992187 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:43.992153    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"]
Sep 16 12:20:43.992939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:43.992905    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d" podUID="1a3c5251-3464-4de5-a39b-348a55da48d8" containerName="payload-processing" containerID="cri-o://b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60" gracePeriod=30
Sep 16 12:20:44.257089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.257033    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:44.301351 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.301319    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c5251-3464-4de5-a39b-348a55da48d8-plugins-config-volume\") pod \"1a3c5251-3464-4de5-a39b-348a55da48d8\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") "
Sep 16 12:20:44.301505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.301489    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z64ds\" (UniqueName: \"kubernetes.io/projected/1a3c5251-3464-4de5-a39b-348a55da48d8-kube-api-access-z64ds\") pod \"1a3c5251-3464-4de5-a39b-348a55da48d8\" (UID: \"1a3c5251-3464-4de5-a39b-348a55da48d8\") "
Sep 16 12:20:44.301654 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.301629    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3c5251-3464-4de5-a39b-348a55da48d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a3c5251-3464-4de5-a39b-348a55da48d8" (UID: "1a3c5251-3464-4de5-a39b-348a55da48d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:44.301861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.301844    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3c5251-3464-4de5-a39b-348a55da48d8-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.304398 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.304365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3c5251-3464-4de5-a39b-348a55da48d8-kube-api-access-z64ds" (OuterVolumeSpecName: "kube-api-access-z64ds") pod "1a3c5251-3464-4de5-a39b-348a55da48d8" (UID: "1a3c5251-3464-4de5-a39b-348a55da48d8"). InnerVolumeSpecName "kube-api-access-z64ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:44.402658 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.402630    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64ds\" (UniqueName: \"kubernetes.io/projected/1a3c5251-3464-4de5-a39b-348a55da48d8-kube-api-access-z64ds\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:20:44.655128 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.655093    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a3c5251-3464-4de5-a39b-348a55da48d8" containerID="b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60" exitCode=0
Sep 16 12:20:44.655288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.655156    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"
Sep 16 12:20:44.655288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.655156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d" event={"ID":"1a3c5251-3464-4de5-a39b-348a55da48d8","Type":"ContainerDied","Data":"b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60"}
Sep 16 12:20:44.655288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.655210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d" event={"ID":"1a3c5251-3464-4de5-a39b-348a55da48d8","Type":"ContainerDied","Data":"927a669881869b72a11804c5eda679a010a88dcb11f9678f24b449634a8a1f78"}
Sep 16 12:20:44.655288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.655249    2560 scope.go:117] "RemoveContainer" containerID="b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60"
Sep 16 12:20:44.664317 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.664296    2560 scope.go:117] "RemoveContainer" containerID="b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60"
Sep 16 12:20:44.664567 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:20:44.664548    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60\": container with ID starting with b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60 not found: ID does not exist" containerID="b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60"
Sep 16 12:20:44.664609 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.664574    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60"} err="failed to get container status \"b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60\": rpc error: code = NotFound desc = could not find container \"b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60\": container with ID starting with b4c2da9ba8e7171e399e6828a0ba5b71cd61d81f3eb2d2b9c64424f894eadf60 not found: ID does not exist"
Sep 16 12:20:44.670955 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.670934    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"]
Sep 16 12:20:44.674427 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:44.674407    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5fb73413-7d6dcc8549-7gb6d"]
Sep 16 12:20:46.326977 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:46.326942    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a3c5251-3464-4de5-a39b-348a55da48d8" path="/var/lib/kubelet/pods/1a3c5251-3464-4de5-a39b-348a55da48d8/volumes"
Sep 16 12:20:53.917978 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.917948    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"]
Sep 16 12:20:53.918376 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.918363    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a3c5251-3464-4de5-a39b-348a55da48d8" containerName="payload-processing"
Sep 16 12:20:53.918376 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.918377    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a3c5251-3464-4de5-a39b-348a55da48d8" containerName="payload-processing"
Sep 16 12:20:53.918453 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.918445    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a3c5251-3464-4de5-a39b-348a55da48d8" containerName="payload-processing"
Sep 16 12:20:53.921401 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.921383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:53.925333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.925299    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"default-dockercfg-cwc4z\""
Sep 16 12:20:53.925454 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.925304    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"simulator-w0-739ee7-kserve-self-signed-certs\""
Sep 16 12:20:53.925454 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.925344    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"kube-root-ca.crt\""
Sep 16 12:20:53.925454 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.925379    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"openshift-service-ca.crt\""
Sep 16 12:20:53.929557 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.929535    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"]
Sep 16 12:20:53.985841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.985817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4r7d\" (UniqueName: \"kubernetes.io/projected/484df775-cdde-4907-b3b1-1a369d374007-kube-api-access-v4r7d\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:53.985932 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.985857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-tmp-dir\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:53.985932 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.985874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484df775-cdde-4907-b3b1-1a369d374007-tls-certs\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:53.986008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.985937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-home\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:53.986008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.985977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-dshm\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:53.986008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:53.986000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-model-cache\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087032 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r7d\" (UniqueName: \"kubernetes.io/projected/484df775-cdde-4907-b3b1-1a369d374007-kube-api-access-v4r7d\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-tmp-dir\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484df775-cdde-4907-b3b1-1a369d374007-tls-certs\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-home\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-dshm\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-model-cache\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087499 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-tmp-dir\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087576 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087561    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-home\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.087632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.087617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-model-cache\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.089441 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.089419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-dshm\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.089695 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.089678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484df775-cdde-4907-b3b1-1a369d374007-tls-certs\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.094472 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.094452    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4r7d\" (UniqueName: \"kubernetes.io/projected/484df775-cdde-4907-b3b1-1a369d374007-kube-api-access-v4r7d\") pod \"simulator-w0-739ee7-kserve-8fc5bbb9-47hdh\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") " pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.231969 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.231905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:20:54.502905 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.502828    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"]
Sep 16 12:20:54.505425 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:20:54.505396    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod484df775_cdde_4907_b3b1_1a369d374007.slice/crio-6e786509dc338860bfd99a54aaed8132f1f42148009729773125059658eae5a2 WatchSource:0}: Error finding container 6e786509dc338860bfd99a54aaed8132f1f42148009729773125059658eae5a2: Status 404 returned error can't find the container with id 6e786509dc338860bfd99a54aaed8132f1f42148009729773125059658eae5a2
Sep 16 12:20:54.708322 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.708279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh" event={"ID":"484df775-cdde-4907-b3b1-1a369d374007","Type":"ContainerStarted","Data":"94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0"}
Sep 16 12:20:54.708466 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.708333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh" event={"ID":"484df775-cdde-4907-b3b1-1a369d374007","Type":"ContainerStarted","Data":"6e786509dc338860bfd99a54aaed8132f1f42148009729773125059658eae5a2"}
Sep 16 12:20:54.727658 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:20:54.727610    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh" podStartSLOduration=1.727594536 podStartE2EDuration="1.727594536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:54.725712995 +0000 UTC m=+1050.861034967" watchObservedRunningTime="2026-09-16 12:20:54.727594536 +0000 UTC m=+1050.862916506"
Sep 16 12:21:04.232469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:04.232438    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:21:04.232929 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:04.232484    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:21:04.244946 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:04.244924    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:21:04.774777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:04.774746    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:21:12.468158 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.468129    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"]
Sep 16 12:21:12.473278 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.473257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"
Sep 16 12:21:12.477316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.477181    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:21:12.477316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.477200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:21:12.479160 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.479134    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"]
Sep 16 12:21:12.554102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.554071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d230fd-88a1-44c9-90fb-cb47737da960-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l\" (UID: \"62d230fd-88a1-44c9-90fb-cb47737da960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"
Sep 16 12:21:12.655511 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.655483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d230fd-88a1-44c9-90fb-cb47737da960-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l\" (UID: \"62d230fd-88a1-44c9-90fb-cb47737da960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"
Sep 16 12:21:12.656084 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.656067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d230fd-88a1-44c9-90fb-cb47737da960-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l\" (UID: \"62d230fd-88a1-44c9-90fb-cb47737da960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"
Sep 16 12:21:12.786650 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:12.786585    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"
Sep 16 12:21:13.061999 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:13.061975    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"]
Sep 16 12:21:13.064263 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:21:13.064236    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d230fd_88a1_44c9_90fb_cb47737da960.slice/crio-7d2ff9fa514a0392abfc0e25f89e7d71048c8f94c6f499a72013894e605ebd57 WatchSource:0}: Error finding container 7d2ff9fa514a0392abfc0e25f89e7d71048c8f94c6f499a72013894e605ebd57: Status 404 returned error can't find the container with id 7d2ff9fa514a0392abfc0e25f89e7d71048c8f94c6f499a72013894e605ebd57
Sep 16 12:21:13.819701 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:13.819659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l" event={"ID":"62d230fd-88a1-44c9-90fb-cb47737da960","Type":"ContainerStarted","Data":"7d2ff9fa514a0392abfc0e25f89e7d71048c8f94c6f499a72013894e605ebd57"}
Sep 16 12:21:15.832587 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:15.832485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l" event={"ID":"62d230fd-88a1-44c9-90fb-cb47737da960","Type":"ContainerStarted","Data":"9020e46df2184ba7458a907eb4169bae170aac460e02f3c5c6b603872f1954fc"}
Sep 16 12:21:15.847374 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:15.847328    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l" podStartSLOduration=1.656558141 podStartE2EDuration="3.84731363s" totalImagesPullingTime="2.190755489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:15.845770556 +0000 UTC m=+1071.981092527" watchObservedRunningTime="2026-09-16 12:21:15.84731363 +0000 UTC m=+1071.982635600"
Sep 16 12:21:16.838986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:16.838957    2560 generic.go:356] "Generic (PLEG): container finished" podID="62d230fd-88a1-44c9-90fb-cb47737da960" containerID="9020e46df2184ba7458a907eb4169bae170aac460e02f3c5c6b603872f1954fc" exitCode=0
Sep 16 12:21:16.839363 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:16.839020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l" event={"ID":"62d230fd-88a1-44c9-90fb-cb47737da960","Type":"ContainerDied","Data":"9020e46df2184ba7458a907eb4169bae170aac460e02f3c5c6b603872f1954fc"}
Sep 16 12:21:17.979411 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:17.979385    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"
Sep 16 12:21:18.103740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:18.103673    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d230fd-88a1-44c9-90fb-cb47737da960-maas-api-service-ca\") pod \"62d230fd-88a1-44c9-90fb-cb47737da960\" (UID: \"62d230fd-88a1-44c9-90fb-cb47737da960\") "
Sep 16 12:21:18.104014 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:18.103992    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d230fd-88a1-44c9-90fb-cb47737da960-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62d230fd-88a1-44c9-90fb-cb47737da960" (UID: "62d230fd-88a1-44c9-90fb-cb47737da960"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:18.205104 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:18.205078    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d230fd-88a1-44c9-90fb-cb47737da960-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:21:18.850337 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:18.850299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l" event={"ID":"62d230fd-88a1-44c9-90fb-cb47737da960","Type":"ContainerDied","Data":"7d2ff9fa514a0392abfc0e25f89e7d71048c8f94c6f499a72013894e605ebd57"}
Sep 16 12:21:18.850337 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:18.850330    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"
Sep 16 12:21:18.850570 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:18.850347    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2ff9fa514a0392abfc0e25f89e7d71048c8f94c6f499a72013894e605ebd57"
Sep 16 12:21:23.911178 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.911147    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"]
Sep 16 12:21:23.911831 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.911811    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d230fd-88a1-44c9-90fb-cb47737da960" containerName="revoke-keys"
Sep 16 12:21:23.911939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.911832    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d230fd-88a1-44c9-90fb-cb47737da960" containerName="revoke-keys"
Sep 16 12:21:23.911997 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.911942    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d230fd-88a1-44c9-90fb-cb47737da960" containerName="revoke-keys"
Sep 16 12:21:23.915149 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.915128    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:23.917691 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.917669    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"openshift-service-ca.crt\""
Sep 16 12:21:23.919007 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.918981    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"simulator-w4-68f7cd-kserve-self-signed-certs\""
Sep 16 12:21:23.919131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.919001    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"kube-root-ca.crt\""
Sep 16 12:21:23.919131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.919056    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"default-dockercfg-4xwdx\""
Sep 16 12:21:23.921648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:23.921625    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"]
Sep 16 12:21:24.057133 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.057109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-tmp-dir\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.057251 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.057142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lk7s\" (UniqueName: \"kubernetes.io/projected/803743e9-b4e7-4d18-986d-f1f221fd2524-kube-api-access-9lk7s\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.057251 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.057189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-home\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.057251 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.057234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803743e9-b4e7-4d18-986d-f1f221fd2524-tls-certs\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.057359 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.057300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-dshm\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.057359 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.057328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-model-cache\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.158720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.158689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-dshm\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.158720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.158722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-model-cache\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.158954 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.158778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-tmp-dir\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.158954 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.158821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lk7s\" (UniqueName: \"kubernetes.io/projected/803743e9-b4e7-4d18-986d-f1f221fd2524-kube-api-access-9lk7s\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.158954 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.158854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-home\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.158954 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.158922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803743e9-b4e7-4d18-986d-f1f221fd2524-tls-certs\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.159213 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.159190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-model-cache\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.159288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.159212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-tmp-dir\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.159288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.159263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-home\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.161152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.161126    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-dshm\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.161409 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.161373    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803743e9-b4e7-4d18-986d-f1f221fd2524-tls-certs\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.166263 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.166239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lk7s\" (UniqueName: \"kubernetes.io/projected/803743e9-b4e7-4d18-986d-f1f221fd2524-kube-api-access-9lk7s\") pod \"simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") " pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.228494 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.228472    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"default-dockercfg-4xwdx\""
Sep 16 12:21:24.236288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.236266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:24.517707 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.517677    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"]
Sep 16 12:21:24.519328 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:21:24.519297    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod803743e9_b4e7_4d18_986d_f1f221fd2524.slice/crio-16e870d33daf194ce18c0e1fa097415cfc497c246f6418bee6d68e3bdc88c64f WatchSource:0}: Error finding container 16e870d33daf194ce18c0e1fa097415cfc497c246f6418bee6d68e3bdc88c64f: Status 404 returned error can't find the container with id 16e870d33daf194ce18c0e1fa097415cfc497c246f6418bee6d68e3bdc88c64f
Sep 16 12:21:24.883381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.883344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc" event={"ID":"803743e9-b4e7-4d18-986d-f1f221fd2524","Type":"ContainerStarted","Data":"4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6"}
Sep 16 12:21:24.883546 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.883395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc" event={"ID":"803743e9-b4e7-4d18-986d-f1f221fd2524","Type":"ContainerStarted","Data":"16e870d33daf194ce18c0e1fa097415cfc497c246f6418bee6d68e3bdc88c64f"}
Sep 16 12:21:24.898714 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:24.898666    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc" podStartSLOduration=1.898652333 podStartE2EDuration="1.898652333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:24.896575793 +0000 UTC m=+1081.031897775" watchObservedRunningTime="2026-09-16 12:21:24.898652333 +0000 UTC m=+1081.033974303"
Sep 16 12:21:28.905782 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:28.905747    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"]
Sep 16 12:21:28.908076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:28.908050    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c69c87e-4cq2l"]
Sep 16 12:21:30.326388 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:30.326352    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d230fd-88a1-44c9-90fb-cb47737da960" path="/var/lib/kubelet/pods/62d230fd-88a1-44c9-90fb-cb47737da960/volumes"
Sep 16 12:21:34.236964 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:34.236931    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:34.237381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:34.236985    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:34.252335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:34.252312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:34.952343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:34.952310    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:21:58.269415 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.269384    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"]
Sep 16 12:21:58.272962 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.272943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"
Sep 16 12:21:58.275468 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.275448    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:21:58.275572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.275491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:21:58.279977 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.279949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"]
Sep 16 12:21:58.366640 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.366613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2560fbaf-87f6-4875-8723-c45e88a5bf1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr\" (UID: \"2560fbaf-87f6-4875-8723-c45e88a5bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"
Sep 16 12:21:58.467842 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.467810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2560fbaf-87f6-4875-8723-c45e88a5bf1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr\" (UID: \"2560fbaf-87f6-4875-8723-c45e88a5bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"
Sep 16 12:21:58.468390 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.468372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2560fbaf-87f6-4875-8723-c45e88a5bf1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr\" (UID: \"2560fbaf-87f6-4875-8723-c45e88a5bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"
Sep 16 12:21:58.586403 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.586346    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"
Sep 16 12:21:58.862247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:58.862224    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"]
Sep 16 12:21:58.864127 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:21:58.864096    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2560fbaf_87f6_4875_8723_c45e88a5bf1b.slice/crio-561cc83b727cfaaa5e6c42250782f31cef645e05d0f3bd20acab6452341b8440 WatchSource:0}: Error finding container 561cc83b727cfaaa5e6c42250782f31cef645e05d0f3bd20acab6452341b8440: Status 404 returned error can't find the container with id 561cc83b727cfaaa5e6c42250782f31cef645e05d0f3bd20acab6452341b8440
Sep 16 12:21:59.072547 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:21:59.072516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr" event={"ID":"2560fbaf-87f6-4875-8723-c45e88a5bf1b","Type":"ContainerStarted","Data":"561cc83b727cfaaa5e6c42250782f31cef645e05d0f3bd20acab6452341b8440"}
Sep 16 12:22:00.080212 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:00.080175    2560 generic.go:356] "Generic (PLEG): container finished" podID="2560fbaf-87f6-4875-8723-c45e88a5bf1b" containerID="9694429967c62a717e9febc3adab949969c277aabfc76512a435a9cb782ee995" exitCode=0
Sep 16 12:22:00.080561 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:00.080223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr" event={"ID":"2560fbaf-87f6-4875-8723-c45e88a5bf1b","Type":"ContainerDied","Data":"9694429967c62a717e9febc3adab949969c277aabfc76512a435a9cb782ee995"}
Sep 16 12:22:01.223425 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:01.223403    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"
Sep 16 12:22:01.290971 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:01.290938    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2560fbaf-87f6-4875-8723-c45e88a5bf1b-maas-api-service-ca\") pod \"2560fbaf-87f6-4875-8723-c45e88a5bf1b\" (UID: \"2560fbaf-87f6-4875-8723-c45e88a5bf1b\") "
Sep 16 12:22:01.291325 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:01.291302    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2560fbaf-87f6-4875-8723-c45e88a5bf1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2560fbaf-87f6-4875-8723-c45e88a5bf1b" (UID: "2560fbaf-87f6-4875-8723-c45e88a5bf1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:01.392054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:01.392020    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2560fbaf-87f6-4875-8723-c45e88a5bf1b-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:22:02.091777 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:02.091746    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"
Sep 16 12:22:02.091967 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:02.091772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr" event={"ID":"2560fbaf-87f6-4875-8723-c45e88a5bf1b","Type":"ContainerDied","Data":"561cc83b727cfaaa5e6c42250782f31cef645e05d0f3bd20acab6452341b8440"}
Sep 16 12:22:02.091967 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:02.091843    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561cc83b727cfaaa5e6c42250782f31cef645e05d0f3bd20acab6452341b8440"
Sep 16 12:22:04.858492 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:04.858454    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"]
Sep 16 12:22:04.862462 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:04.862435    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f9d3de8-cb8lr"]
Sep 16 12:22:06.325446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:06.325414    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2560fbaf-87f6-4875-8723-c45e88a5bf1b" path="/var/lib/kubelet/pods/2560fbaf-87f6-4875-8723-c45e88a5bf1b/volumes"
Sep 16 12:22:20.479712 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.479675    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"]
Sep 16 12:22:20.480160 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.480147    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2560fbaf-87f6-4875-8723-c45e88a5bf1b" containerName="revoke-keys"
Sep 16 12:22:20.480160 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.480162    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2560fbaf-87f6-4875-8723-c45e88a5bf1b" containerName="revoke-keys"
Sep 16 12:22:20.480247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.480238    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2560fbaf-87f6-4875-8723-c45e88a5bf1b" containerName="revoke-keys"
Sep 16 12:22:20.483595 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.483570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"
Sep 16 12:22:20.485968 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.485946    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:22:20.485968 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.485962    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:22:20.491184 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.491161    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"]
Sep 16 12:22:20.575832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.575801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/764fa6a9-d290-4a5b-a992-93bbbcaf59bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-00a141-qct7p\" (UID: \"764fa6a9-d290-4a5b-a992-93bbbcaf59bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"
Sep 16 12:22:20.677340 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.677309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/764fa6a9-d290-4a5b-a992-93bbbcaf59bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-00a141-qct7p\" (UID: \"764fa6a9-d290-4a5b-a992-93bbbcaf59bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"
Sep 16 12:22:20.677940 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.677920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/764fa6a9-d290-4a5b-a992-93bbbcaf59bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-00a141-qct7p\" (UID: \"764fa6a9-d290-4a5b-a992-93bbbcaf59bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"
Sep 16 12:22:20.797714 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:20.797622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"
Sep 16 12:22:21.070815 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:21.070776    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"]
Sep 16 12:22:21.073051 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:22:21.073025    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764fa6a9_d290_4a5b_a992_93bbbcaf59bb.slice/crio-0cacb237aeaa4a2352bab699bd179ec4f894fd834224e8c5510a8066bee1d6f8 WatchSource:0}: Error finding container 0cacb237aeaa4a2352bab699bd179ec4f894fd834224e8c5510a8066bee1d6f8: Status 404 returned error can't find the container with id 0cacb237aeaa4a2352bab699bd179ec4f894fd834224e8c5510a8066bee1d6f8
Sep 16 12:22:21.195326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:21.195288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p" event={"ID":"764fa6a9-d290-4a5b-a992-93bbbcaf59bb","Type":"ContainerStarted","Data":"0cacb237aeaa4a2352bab699bd179ec4f894fd834224e8c5510a8066bee1d6f8"}
Sep 16 12:22:22.202677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:22.202642    2560 generic.go:356] "Generic (PLEG): container finished" podID="764fa6a9-d290-4a5b-a992-93bbbcaf59bb" containerID="4ae3266b53f07ed15c39d0b7969bcfabc6f5ae6e2d518526ce35cd881d4742e0" exitCode=0
Sep 16 12:22:22.203112 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:22.202744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p" event={"ID":"764fa6a9-d290-4a5b-a992-93bbbcaf59bb","Type":"ContainerDied","Data":"4ae3266b53f07ed15c39d0b7969bcfabc6f5ae6e2d518526ce35cd881d4742e0"}
Sep 16 12:22:23.350740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:23.350718    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"
Sep 16 12:22:23.405134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:23.405103    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/764fa6a9-d290-4a5b-a992-93bbbcaf59bb-maas-api-service-ca\") pod \"764fa6a9-d290-4a5b-a992-93bbbcaf59bb\" (UID: \"764fa6a9-d290-4a5b-a992-93bbbcaf59bb\") "
Sep 16 12:22:23.405445 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:23.405425    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764fa6a9-d290-4a5b-a992-93bbbcaf59bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "764fa6a9-d290-4a5b-a992-93bbbcaf59bb" (UID: "764fa6a9-d290-4a5b-a992-93bbbcaf59bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:23.506351 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:23.506270    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/764fa6a9-d290-4a5b-a992-93bbbcaf59bb-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:22:24.216022 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:24.215991    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"
Sep 16 12:22:24.216287 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:24.216030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p" event={"ID":"764fa6a9-d290-4a5b-a992-93bbbcaf59bb","Type":"ContainerDied","Data":"0cacb237aeaa4a2352bab699bd179ec4f894fd834224e8c5510a8066bee1d6f8"}
Sep 16 12:22:24.216287 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:24.216082    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cacb237aeaa4a2352bab699bd179ec4f894fd834224e8c5510a8066bee1d6f8"
Sep 16 12:22:28.528853 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.528816    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"]
Sep 16 12:22:28.529266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.529253    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764fa6a9-d290-4a5b-a992-93bbbcaf59bb" containerName="revoke-keys"
Sep 16 12:22:28.529266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.529267    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="764fa6a9-d290-4a5b-a992-93bbbcaf59bb" containerName="revoke-keys"
Sep 16 12:22:28.529345 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.529338    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="764fa6a9-d290-4a5b-a992-93bbbcaf59bb" containerName="revoke-keys"
Sep 16 12:22:28.532567 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.532551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"
Sep 16 12:22:28.535030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.535009    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:22:28.535154 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.535074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:22:28.539531 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.539505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"]
Sep 16 12:22:28.654604 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.654573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4af823-48af-4012-bf76-3775e1e24fd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx\" (UID: \"4e4af823-48af-4012-bf76-3775e1e24fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"
Sep 16 12:22:28.755957 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.755925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4af823-48af-4012-bf76-3775e1e24fd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx\" (UID: \"4e4af823-48af-4012-bf76-3775e1e24fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"
Sep 16 12:22:28.756500 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.756482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4af823-48af-4012-bf76-3775e1e24fd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx\" (UID: \"4e4af823-48af-4012-bf76-3775e1e24fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"
Sep 16 12:22:28.844909 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:28.844877    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"
Sep 16 12:22:29.116316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:29.116218    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"]
Sep 16 12:22:29.118694 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:22:29.118668    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4af823_48af_4012_bf76_3775e1e24fd2.slice/crio-36e66aea50b81aa504117c65f1544af5aa4c3981cbb0f38c060c6ff96b20974d WatchSource:0}: Error finding container 36e66aea50b81aa504117c65f1544af5aa4c3981cbb0f38c060c6ff96b20974d: Status 404 returned error can't find the container with id 36e66aea50b81aa504117c65f1544af5aa4c3981cbb0f38c060c6ff96b20974d
Sep 16 12:22:29.244056 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:29.244026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx" event={"ID":"4e4af823-48af-4012-bf76-3775e1e24fd2","Type":"ContainerStarted","Data":"36e66aea50b81aa504117c65f1544af5aa4c3981cbb0f38c060c6ff96b20974d"}
Sep 16 12:22:30.251324 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:30.251292    2560 generic.go:356] "Generic (PLEG): container finished" podID="4e4af823-48af-4012-bf76-3775e1e24fd2" containerID="02f5824f41b2e46b33c4d32bac1651f9263f4ba130598d6b05c5f2a1c1c7d760" exitCode=0
Sep 16 12:22:30.251669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:30.251374    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx" event={"ID":"4e4af823-48af-4012-bf76-3775e1e24fd2","Type":"ContainerDied","Data":"02f5824f41b2e46b33c4d32bac1651f9263f4ba130598d6b05c5f2a1c1c7d760"}
Sep 16 12:22:31.403429 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:31.403404    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"
Sep 16 12:22:31.481912 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:31.481885    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4af823-48af-4012-bf76-3775e1e24fd2-maas-api-service-ca\") pod \"4e4af823-48af-4012-bf76-3775e1e24fd2\" (UID: \"4e4af823-48af-4012-bf76-3775e1e24fd2\") "
Sep 16 12:22:31.482254 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:31.482234    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4af823-48af-4012-bf76-3775e1e24fd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e4af823-48af-4012-bf76-3775e1e24fd2" (UID: "4e4af823-48af-4012-bf76-3775e1e24fd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:31.583574 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:31.583514    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4af823-48af-4012-bf76-3775e1e24fd2-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:22:32.263968 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:32.263944    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"
Sep 16 12:22:32.263968 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:32.263948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx" event={"ID":"4e4af823-48af-4012-bf76-3775e1e24fd2","Type":"ContainerDied","Data":"36e66aea50b81aa504117c65f1544af5aa4c3981cbb0f38c060c6ff96b20974d"}
Sep 16 12:22:32.264215 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:32.263991    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e66aea50b81aa504117c65f1544af5aa4c3981cbb0f38c060c6ff96b20974d"
Sep 16 12:22:32.990856 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:32.990819    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"]
Sep 16 12:22:32.991316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:32.991035    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr" podUID="eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" containerName="payload-processing" containerID="cri-o://2ddc49ce9ee19f31f586c4276f96647b121e10e7721bdfd8ecc01e32ff5c13b2" gracePeriod=30
Sep 16 12:22:32.996942 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:32.996918    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"]
Sep 16 12:22:32.997112 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:32.997085    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j" podUID="3d0f7950-9a2d-4834-abe6-95914cec587d" containerName="payload-pre-processing" containerID="cri-o://0473b2bd40587e5174dc60211d4067cec8a6dcce3bcec9876ac1535dc8f0a6e0" gracePeriod=30
Sep 16 12:22:33.273186 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.273108    2560 generic.go:356] "Generic (PLEG): container finished" podID="eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" containerID="2ddc49ce9ee19f31f586c4276f96647b121e10e7721bdfd8ecc01e32ff5c13b2" exitCode=0
Sep 16 12:22:33.273329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.273178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr" event={"ID":"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae","Type":"ContainerDied","Data":"2ddc49ce9ee19f31f586c4276f96647b121e10e7721bdfd8ecc01e32ff5c13b2"}
Sep 16 12:22:33.275140 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.275114    2560 generic.go:356] "Generic (PLEG): container finished" podID="3d0f7950-9a2d-4834-abe6-95914cec587d" containerID="0473b2bd40587e5174dc60211d4067cec8a6dcce3bcec9876ac1535dc8f0a6e0" exitCode=0
Sep 16 12:22:33.275263 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.275191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j" event={"ID":"3d0f7950-9a2d-4834-abe6-95914cec587d","Type":"ContainerDied","Data":"0473b2bd40587e5174dc60211d4067cec8a6dcce3bcec9876ac1535dc8f0a6e0"}
Sep 16 12:22:33.290214 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.290191    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"]
Sep 16 12:22:33.292843 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.292823    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-00a141-qct7p"]
Sep 16 12:22:33.303401 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.303381    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:22:33.306547 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.306529    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:22:33.400872 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.400843    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-plugins-config-volume\") pod \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") "
Sep 16 12:22:33.401030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.400897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2dfz\" (UniqueName: \"kubernetes.io/projected/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-kube-api-access-x2dfz\") pod \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\" (UID: \"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae\") "
Sep 16 12:22:33.401030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.400936    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f2vt\" (UniqueName: \"kubernetes.io/projected/3d0f7950-9a2d-4834-abe6-95914cec587d-kube-api-access-7f2vt\") pod \"3d0f7950-9a2d-4834-abe6-95914cec587d\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") "
Sep 16 12:22:33.401030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.400956    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d0f7950-9a2d-4834-abe6-95914cec587d-plugins-config-volume\") pod \"3d0f7950-9a2d-4834-abe6-95914cec587d\" (UID: \"3d0f7950-9a2d-4834-abe6-95914cec587d\") "
Sep 16 12:22:33.401205 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.401177    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" (UID: "eb17a61a-9daa-44cb-ba18-fdce6a08c9ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:33.401332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.401311    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:22:33.401490 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.401409    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0f7950-9a2d-4834-abe6-95914cec587d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d0f7950-9a2d-4834-abe6-95914cec587d" (UID: "3d0f7950-9a2d-4834-abe6-95914cec587d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:33.403218 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.403194    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-kube-api-access-x2dfz" (OuterVolumeSpecName: "kube-api-access-x2dfz") pod "eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" (UID: "eb17a61a-9daa-44cb-ba18-fdce6a08c9ae"). InnerVolumeSpecName "kube-api-access-x2dfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:33.403316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.403248    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0f7950-9a2d-4834-abe6-95914cec587d-kube-api-access-7f2vt" (OuterVolumeSpecName: "kube-api-access-7f2vt") pod "3d0f7950-9a2d-4834-abe6-95914cec587d" (UID: "3d0f7950-9a2d-4834-abe6-95914cec587d"). InnerVolumeSpecName "kube-api-access-7f2vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:33.502730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.502705    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f2vt\" (UniqueName: \"kubernetes.io/projected/3d0f7950-9a2d-4834-abe6-95914cec587d-kube-api-access-7f2vt\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:22:33.502730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.502730    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d0f7950-9a2d-4834-abe6-95914cec587d-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:22:33.502913 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:33.502744    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2dfz\" (UniqueName: \"kubernetes.io/projected/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae-kube-api-access-x2dfz\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:22:34.282145 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.282114    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"
Sep 16 12:22:34.282145 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.282136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr" event={"ID":"eb17a61a-9daa-44cb-ba18-fdce6a08c9ae","Type":"ContainerDied","Data":"15e72c3c31a96d5f1343ac662365260db0a3c5b34195cb2eefcedce5430ac289"}
Sep 16 12:22:34.282655 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.282200    2560 scope.go:117] "RemoveContainer" containerID="2ddc49ce9ee19f31f586c4276f96647b121e10e7721bdfd8ecc01e32ff5c13b2"
Sep 16 12:22:34.283833 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.283778    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"
Sep 16 12:22:34.283958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.283839    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j" event={"ID":"3d0f7950-9a2d-4834-abe6-95914cec587d","Type":"ContainerDied","Data":"d711984d5de3490d168d785490a0c3932f628626a81ff03a37fba8edc4483565"}
Sep 16 12:22:34.293720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.293593    2560 scope.go:117] "RemoveContainer" containerID="0473b2bd40587e5174dc60211d4067cec8a6dcce3bcec9876ac1535dc8f0a6e0"
Sep 16 12:22:34.315131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.315106    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"]
Sep 16 12:22:34.317051 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.317028    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-00a141-566ccbb6c5-q8zfr"]
Sep 16 12:22:34.325641 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.325614    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="764fa6a9-d290-4a5b-a992-93bbbcaf59bb" path="/var/lib/kubelet/pods/764fa6a9-d290-4a5b-a992-93bbbcaf59bb/volumes"
Sep 16 12:22:34.326056 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.326036    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" path="/var/lib/kubelet/pods/eb17a61a-9daa-44cb-ba18-fdce6a08c9ae/volumes"
Sep 16 12:22:34.326559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.326540    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"]
Sep 16 12:22:34.332960 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:34.332934    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-00a141-57f8c95cfd-7dz9j"]
Sep 16 12:22:36.325290 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:36.325258    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0f7950-9a2d-4834-abe6-95914cec587d" path="/var/lib/kubelet/pods/3d0f7950-9a2d-4834-abe6-95914cec587d/volumes"
Sep 16 12:22:39.453605 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:39.453573    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"]
Sep 16 12:22:39.462003 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:39.457883    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2aae110-bm9mx"]
Sep 16 12:22:40.324068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:22:40.324033    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4af823-48af-4012-bf76-3775e1e24fd2" path="/var/lib/kubelet/pods/4e4af823-48af-4012-bf76-3775e1e24fd2/volumes"
Sep 16 12:23:24.345749 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:23:24.345720    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:23:24.347681 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:23:24.347656    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:24:08.431821 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.431755    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"]
Sep 16 12:24:08.432460 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432434    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4af823-48af-4012-bf76-3775e1e24fd2" containerName="revoke-keys"
Sep 16 12:24:08.432460 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432470    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4af823-48af-4012-bf76-3775e1e24fd2" containerName="revoke-keys"
Sep 16 12:24:08.432603 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432533    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0f7950-9a2d-4834-abe6-95914cec587d" containerName="payload-pre-processing"
Sep 16 12:24:08.432603 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432542    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0f7950-9a2d-4834-abe6-95914cec587d" containerName="payload-pre-processing"
Sep 16 12:24:08.432603 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432565    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" containerName="payload-processing"
Sep 16 12:24:08.432603 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432573    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" containerName="payload-processing"
Sep 16 12:24:08.432747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432679    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0f7950-9a2d-4834-abe6-95914cec587d" containerName="payload-pre-processing"
Sep 16 12:24:08.432747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432694    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4af823-48af-4012-bf76-3775e1e24fd2" containerName="revoke-keys"
Sep 16 12:24:08.432747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.432705    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb17a61a-9daa-44cb-ba18-fdce6a08c9ae" containerName="payload-processing"
Sep 16 12:24:08.436359 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.436338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.438950 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.438932    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"premium-w4-68f7cd-kserve-self-signed-certs\""
Sep 16 12:24:08.447060 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.447020    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"]
Sep 16 12:24:08.532707 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.532686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3aa3376-682f-4569-ada5-bd79f40d5c88-tls-certs\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.532857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.532713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-tmp-dir\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.532857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.532766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f47r\" (UniqueName: \"kubernetes.io/projected/b3aa3376-682f-4569-ada5-bd79f40d5c88-kube-api-access-7f47r\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.532857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.532835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-dshm\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.532985 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.532871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-home\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.532985 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.532913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-model-cache\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.633978 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.633958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3aa3376-682f-4569-ada5-bd79f40d5c88-tls-certs\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.633986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-tmp-dir\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.634031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f47r\" (UniqueName: \"kubernetes.io/projected/b3aa3376-682f-4569-ada5-bd79f40d5c88-kube-api-access-7f47r\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634233 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.634214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-dshm\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634306 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.634283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-home\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634361 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.634317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-model-cache\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634466 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.634449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-tmp-dir\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634646 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.634621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-home\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.634726 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.634682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-model-cache\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.636460 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.636432    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-dshm\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.636693 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.636675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3aa3376-682f-4569-ada5-bd79f40d5c88-tls-certs\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.643473 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.643453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f47r\" (UniqueName: \"kubernetes.io/projected/b3aa3376-682f-4569-ada5-bd79f40d5c88-kube-api-access-7f47r\") pod \"premium-w4-68f7cd-kserve-b46775474-b4jc4\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") " pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:08.748711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:08.748648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:09.024952 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:09.024927    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"]
Sep 16 12:24:09.027509 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:24:09.027483    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3aa3376_682f_4569_ada5_bd79f40d5c88.slice/crio-8ce3a2a990463c346722904bfef53f9261c3cf1bccae7a6c04c408c35a5812aa WatchSource:0}: Error finding container 8ce3a2a990463c346722904bfef53f9261c3cf1bccae7a6c04c408c35a5812aa: Status 404 returned error can't find the container with id 8ce3a2a990463c346722904bfef53f9261c3cf1bccae7a6c04c408c35a5812aa
Sep 16 12:24:09.795503 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:09.795467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4" event={"ID":"b3aa3376-682f-4569-ada5-bd79f40d5c88","Type":"ContainerStarted","Data":"88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb"}
Sep 16 12:24:09.795503 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:09.795505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4" event={"ID":"b3aa3376-682f-4569-ada5-bd79f40d5c88","Type":"ContainerStarted","Data":"8ce3a2a990463c346722904bfef53f9261c3cf1bccae7a6c04c408c35a5812aa"}
Sep 16 12:24:09.819997 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:09.816637    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4" podStartSLOduration=1.8166168200000001 podStartE2EDuration="1.81661682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:09.813948196 +0000 UTC m=+1245.949270168" watchObservedRunningTime="2026-09-16 12:24:09.81661682 +0000 UTC m=+1245.951938792"
Sep 16 12:24:18.749330 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:18.749249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:18.749330 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:18.749312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:18.761494 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:18.761469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:18.857476 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:18.857443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:24:29.961963 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:29.961924    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"]
Sep 16 12:24:29.965875 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:29.965853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:29.968442 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:29.968413    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cadab1d8\""
Sep 16 12:24:29.968553 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:29.968413    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cadab1d8-dockercfg-w4ngh\""
Sep 16 12:24:29.975962 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:29.975938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"]
Sep 16 12:24:30.140353 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.140316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") " pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.140506 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.140376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdg5\" (UniqueName: \"kubernetes.io/projected/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-kube-api-access-kqdg5\") pod \"payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") " pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.241108 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.241035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") " pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.241108 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.241077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdg5\" (UniqueName: \"kubernetes.io/projected/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-kube-api-access-kqdg5\") pod \"payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") " pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.241711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.241693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-plugins-config-volume\") pod \"payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") " pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.249308 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.249276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdg5\" (UniqueName: \"kubernetes.io/projected/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-kube-api-access-kqdg5\") pod \"payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") " pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.276995 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.276974    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.548772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.548745    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"]
Sep 16 12:24:30.550523 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:24:30.550498    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69831554_2e34_4eef_a1f0_1ed9e4f6cb2d.slice/crio-9663052d55e7c50f7256f5cb338360b8437a69ff360ccaf7ae36e15ccc0b1303 WatchSource:0}: Error finding container 9663052d55e7c50f7256f5cb338360b8437a69ff360ccaf7ae36e15ccc0b1303: Status 404 returned error can't find the container with id 9663052d55e7c50f7256f5cb338360b8437a69ff360ccaf7ae36e15ccc0b1303
Sep 16 12:24:30.918694 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.918659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4" event={"ID":"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d","Type":"ContainerStarted","Data":"002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0"}
Sep 16 12:24:30.918883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.918709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4" event={"ID":"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d","Type":"ContainerStarted","Data":"9663052d55e7c50f7256f5cb338360b8437a69ff360ccaf7ae36e15ccc0b1303"}
Sep 16 12:24:30.918883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.918803    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:24:30.934958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:30.934911    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4" podStartSLOduration=1.934896545 podStartE2EDuration="1.934896545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:30.93400989 +0000 UTC m=+1267.069331863" watchObservedRunningTime="2026-09-16 12:24:30.934896545 +0000 UTC m=+1267.070218516"
Sep 16 12:24:41.925781 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:24:41.925752    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:25:03.054780 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.054746    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"]
Sep 16 12:25:03.058009 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.057991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"
Sep 16 12:25:03.060594 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.060571    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:25:03.060735 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.060718    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:25:03.066542 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.066519    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"]
Sep 16 12:25:03.129694 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.129668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783eb05c-5486-4a9f-9015-b26f265a2e3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm\" (UID: \"783eb05c-5486-4a9f-9015-b26f265a2e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"
Sep 16 12:25:03.230384 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.230352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783eb05c-5486-4a9f-9015-b26f265a2e3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm\" (UID: \"783eb05c-5486-4a9f-9015-b26f265a2e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"
Sep 16 12:25:03.230939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.230920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783eb05c-5486-4a9f-9015-b26f265a2e3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm\" (UID: \"783eb05c-5486-4a9f-9015-b26f265a2e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"
Sep 16 12:25:03.370155 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.370134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"
Sep 16 12:25:03.642544 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:03.642521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"]
Sep 16 12:25:03.644273 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:25:03.644246    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783eb05c_5486_4a9f_9015_b26f265a2e3e.slice/crio-b5a63fc9783339b7dfe7dbef63bdedd91a254544094f36a54c5271882a4c161a WatchSource:0}: Error finding container b5a63fc9783339b7dfe7dbef63bdedd91a254544094f36a54c5271882a4c161a: Status 404 returned error can't find the container with id b5a63fc9783339b7dfe7dbef63bdedd91a254544094f36a54c5271882a4c161a
Sep 16 12:25:04.102172 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:04.102137    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm" event={"ID":"783eb05c-5486-4a9f-9015-b26f265a2e3e","Type":"ContainerStarted","Data":"b06ee201382c1b4268aa961969463cd3e0a8441b1671c5db0669de4e43b4c929"}
Sep 16 12:25:04.102503 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:04.102200    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm" event={"ID":"783eb05c-5486-4a9f-9015-b26f265a2e3e","Type":"ContainerStarted","Data":"b5a63fc9783339b7dfe7dbef63bdedd91a254544094f36a54c5271882a4c161a"}
Sep 16 12:25:04.115719 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:04.115666    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm" podStartSLOduration=1.115646795 podStartE2EDuration="1.115646795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:04.114842593 +0000 UTC m=+1300.250164564" watchObservedRunningTime="2026-09-16 12:25:04.115646795 +0000 UTC m=+1300.250968767"
Sep 16 12:25:05.109631 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:05.109597    2560 generic.go:356] "Generic (PLEG): container finished" podID="783eb05c-5486-4a9f-9015-b26f265a2e3e" containerID="b06ee201382c1b4268aa961969463cd3e0a8441b1671c5db0669de4e43b4c929" exitCode=0
Sep 16 12:25:05.110027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:05.109686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm" event={"ID":"783eb05c-5486-4a9f-9015-b26f265a2e3e","Type":"ContainerDied","Data":"b06ee201382c1b4268aa961969463cd3e0a8441b1671c5db0669de4e43b4c929"}
Sep 16 12:25:06.253680 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:06.253659    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"
Sep 16 12:25:06.360273 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:06.360249    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783eb05c-5486-4a9f-9015-b26f265a2e3e-maas-api-service-ca\") pod \"783eb05c-5486-4a9f-9015-b26f265a2e3e\" (UID: \"783eb05c-5486-4a9f-9015-b26f265a2e3e\") "
Sep 16 12:25:06.360596 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:06.360572    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783eb05c-5486-4a9f-9015-b26f265a2e3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "783eb05c-5486-4a9f-9015-b26f265a2e3e" (UID: "783eb05c-5486-4a9f-9015-b26f265a2e3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:06.461589 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:06.461530    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/783eb05c-5486-4a9f-9015-b26f265a2e3e-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:25:07.129233 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:07.129196    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"
Sep 16 12:25:07.129233 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:07.129216    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm" event={"ID":"783eb05c-5486-4a9f-9015-b26f265a2e3e","Type":"ContainerDied","Data":"b5a63fc9783339b7dfe7dbef63bdedd91a254544094f36a54c5271882a4c161a"}
Sep 16 12:25:07.129446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:07.129289    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5a63fc9783339b7dfe7dbef63bdedd91a254544094f36a54c5271882a4c161a"
Sep 16 12:25:12.652333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.652306    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"]
Sep 16 12:25:12.652984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.652724    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783eb05c-5486-4a9f-9015-b26f265a2e3e" containerName="revoke-keys"
Sep 16 12:25:12.652984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.652735    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="783eb05c-5486-4a9f-9015-b26f265a2e3e" containerName="revoke-keys"
Sep 16 12:25:12.652984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.652817    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="783eb05c-5486-4a9f-9015-b26f265a2e3e" containerName="revoke-keys"
Sep 16 12:25:12.656003 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.655980    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"
Sep 16 12:25:12.658522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.658494    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:25:12.658641 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.658502    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:25:12.663177 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.663150    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"]
Sep 16 12:25:12.820955 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.820921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e609ce-940c-4441-aef0-0de84f54eedd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7\" (UID: \"e1e609ce-940c-4441-aef0-0de84f54eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"
Sep 16 12:25:12.921912 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.921812    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e609ce-940c-4441-aef0-0de84f54eedd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7\" (UID: \"e1e609ce-940c-4441-aef0-0de84f54eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"
Sep 16 12:25:12.922497 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.922471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e609ce-940c-4441-aef0-0de84f54eedd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7\" (UID: \"e1e609ce-940c-4441-aef0-0de84f54eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"
Sep 16 12:25:12.968933 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:12.968911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"
Sep 16 12:25:13.238115 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.238086    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"]
Sep 16 12:25:13.240262 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:25:13.240224    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e609ce_940c_4441_aef0_0de84f54eedd.slice/crio-6a99c0133c4a93d05d3ee7464105d9d40ca9e5be3c0f80dbb886c25bd5700409 WatchSource:0}: Error finding container 6a99c0133c4a93d05d3ee7464105d9d40ca9e5be3c0f80dbb886c25bd5700409: Status 404 returned error can't find the container with id 6a99c0133c4a93d05d3ee7464105d9d40ca9e5be3c0f80dbb886c25bd5700409
Sep 16 12:25:13.453442 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.453199    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"]
Sep 16 12:25:13.453641 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.453546    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4" podUID="69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" containerName="payload-processing" containerID="cri-o://002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0" gracePeriod=30
Sep 16 12:25:13.597582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.597532    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"]
Sep 16 12:25:13.599699 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.599672    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cadab1d8-8xsjm"]
Sep 16 12:25:13.694397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.694374    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:25:13.832026 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.831955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqdg5\" (UniqueName: \"kubernetes.io/projected/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-kube-api-access-kqdg5\") pod \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") "
Sep 16 12:25:13.832026 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.832007    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-plugins-config-volume\") pod \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\" (UID: \"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d\") "
Sep 16 12:25:13.832369 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.832348    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" (UID: "69831554-2e34-4eef-a1f0-1ed9e4f6cb2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:13.834259 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.834239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-kube-api-access-kqdg5" (OuterVolumeSpecName: "kube-api-access-kqdg5") pod "69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" (UID: "69831554-2e34-4eef-a1f0-1ed9e4f6cb2d"). InnerVolumeSpecName "kube-api-access-kqdg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:13.932767 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.932741    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqdg5\" (UniqueName: \"kubernetes.io/projected/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-kube-api-access-kqdg5\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.932767 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:13.932764    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:25:14.169140 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.169112    2560 generic.go:356] "Generic (PLEG): container finished" podID="69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" containerID="002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0" exitCode=0
Sep 16 12:25:14.169288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.169159    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"
Sep 16 12:25:14.169288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.169189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4" event={"ID":"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d","Type":"ContainerDied","Data":"002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0"}
Sep 16 12:25:14.169288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.169244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4" event={"ID":"69831554-2e34-4eef-a1f0-1ed9e4f6cb2d","Type":"ContainerDied","Data":"9663052d55e7c50f7256f5cb338360b8437a69ff360ccaf7ae36e15ccc0b1303"}
Sep 16 12:25:14.169288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.169266    2560 scope.go:117] "RemoveContainer" containerID="002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0"
Sep 16 12:25:14.171088 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.171068    2560 generic.go:356] "Generic (PLEG): container finished" podID="e1e609ce-940c-4441-aef0-0de84f54eedd" containerID="32f35805331afac4230b30fd87acc5487d77ff1177103fc0d62e8844664350bb" exitCode=0
Sep 16 12:25:14.171187 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.171161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7" event={"ID":"e1e609ce-940c-4441-aef0-0de84f54eedd","Type":"ContainerDied","Data":"32f35805331afac4230b30fd87acc5487d77ff1177103fc0d62e8844664350bb"}
Sep 16 12:25:14.171250 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.171213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7" event={"ID":"e1e609ce-940c-4441-aef0-0de84f54eedd","Type":"ContainerStarted","Data":"6a99c0133c4a93d05d3ee7464105d9d40ca9e5be3c0f80dbb886c25bd5700409"}
Sep 16 12:25:14.178559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.178545    2560 scope.go:117] "RemoveContainer" containerID="002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0"
Sep 16 12:25:14.178796 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:25:14.178766    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0\": container with ID starting with 002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0 not found: ID does not exist" containerID="002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0"
Sep 16 12:25:14.178849 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.178807    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0"} err="failed to get container status \"002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0\": rpc error: code = NotFound desc = could not find container \"002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0\": container with ID starting with 002894235ae11dcd7455f0630456896934110dc42bbfc71c0b894926a6c7bac0 not found: ID does not exist"
Sep 16 12:25:14.200013 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.199991    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"]
Sep 16 12:25:14.203809 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.203772    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cadab1d8-77459fd95c-rx7g4"]
Sep 16 12:25:14.325070 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.325044    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" path="/var/lib/kubelet/pods/69831554-2e34-4eef-a1f0-1ed9e4f6cb2d/volumes"
Sep 16 12:25:14.325445 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:14.325426    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783eb05c-5486-4a9f-9015-b26f265a2e3e" path="/var/lib/kubelet/pods/783eb05c-5486-4a9f-9015-b26f265a2e3e/volumes"
Sep 16 12:25:15.315933 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:15.315910    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"
Sep 16 12:25:15.446122 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:15.446037    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e609ce-940c-4441-aef0-0de84f54eedd-maas-api-service-ca\") pod \"e1e609ce-940c-4441-aef0-0de84f54eedd\" (UID: \"e1e609ce-940c-4441-aef0-0de84f54eedd\") "
Sep 16 12:25:15.446396 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:15.446373    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e609ce-940c-4441-aef0-0de84f54eedd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1e609ce-940c-4441-aef0-0de84f54eedd" (UID: "e1e609ce-940c-4441-aef0-0de84f54eedd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:15.446549 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:15.446514    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e609ce-940c-4441-aef0-0de84f54eedd-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:25:16.184193 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:16.184159    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"
Sep 16 12:25:16.184357 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:16.184208    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7" event={"ID":"e1e609ce-940c-4441-aef0-0de84f54eedd","Type":"ContainerDied","Data":"6a99c0133c4a93d05d3ee7464105d9d40ca9e5be3c0f80dbb886c25bd5700409"}
Sep 16 12:25:16.184357 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:16.184263    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a99c0133c4a93d05d3ee7464105d9d40ca9e5be3c0f80dbb886c25bd5700409"
Sep 16 12:25:23.549445 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:23.549408    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"]
Sep 16 12:25:23.552280 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:23.552253    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f1730c-4xzp7"]
Sep 16 12:25:24.325859 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:24.325830    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e609ce-940c-4441-aef0-0de84f54eedd" path="/var/lib/kubelet/pods/e1e609ce-940c-4441-aef0-0de84f54eedd/volumes"
Sep 16 12:25:54.119266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.119184    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"]
Sep 16 12:25:54.119705 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.119616    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e609ce-940c-4441-aef0-0de84f54eedd" containerName="revoke-keys"
Sep 16 12:25:54.119705 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.119629    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e609ce-940c-4441-aef0-0de84f54eedd" containerName="revoke-keys"
Sep 16 12:25:54.119705 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.119659    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" containerName="payload-processing"
Sep 16 12:25:54.119705 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.119667    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" containerName="payload-processing"
Sep 16 12:25:54.119877 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.119739    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="69831554-2e34-4eef-a1f0-1ed9e4f6cb2d" containerName="payload-processing"
Sep 16 12:25:54.119877 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.119751    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e609ce-940c-4441-aef0-0de84f54eedd" containerName="revoke-keys"
Sep 16 12:25:54.122939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.122920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.125432 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.125411    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"e2e-conflict-a-786e64f4-kserve-self-signed-certs\""
Sep 16 12:25:54.132039 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.132017    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"]
Sep 16 12:25:54.282213 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.282182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5gx5\" (UniqueName: \"kubernetes.io/projected/4b05d9a5-7018-4691-b5d7-c2c274c5109d-kube-api-access-w5gx5\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.282350 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.282218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tls-certs\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.282402 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.282348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-home\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.282402 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.282373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-model-cache\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.282402 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.282396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-dshm\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.282572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.282425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tmp-dir\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.383720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.383657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5gx5\" (UniqueName: \"kubernetes.io/projected/4b05d9a5-7018-4691-b5d7-c2c274c5109d-kube-api-access-w5gx5\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.383720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.383689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tls-certs\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.383889 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.383852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-home\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.383889 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.383880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-model-cache\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.383970 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.383903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-dshm\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.384028 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.383988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tmp-dir\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.384226 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.384199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-home\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.384343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.384326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-model-cache\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.384566 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.384395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tmp-dir\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.386191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.386163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-dshm\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.386370 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.386351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tls-certs\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.390780 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.390761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5gx5\" (UniqueName: \"kubernetes.io/projected/4b05d9a5-7018-4691-b5d7-c2c274c5109d-kube-api-access-w5gx5\") pod \"e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") " pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.434102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.434082    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:25:54.707743 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:54.707711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"]
Sep 16 12:25:54.710084 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:25:54.710058    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b05d9a5_7018_4691_b5d7_c2c274c5109d.slice/crio-7057ceb1f3461347f0f48c8a26a2dc2b1757646048222fadd2ee9025d8d6e3ef WatchSource:0}: Error finding container 7057ceb1f3461347f0f48c8a26a2dc2b1757646048222fadd2ee9025d8d6e3ef: Status 404 returned error can't find the container with id 7057ceb1f3461347f0f48c8a26a2dc2b1757646048222fadd2ee9025d8d6e3ef
Sep 16 12:25:55.409065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:55.409036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2" event={"ID":"4b05d9a5-7018-4691-b5d7-c2c274c5109d","Type":"ContainerStarted","Data":"9cc1618ec35f5023ac9ecc2ea6df297800f40ff7f3dda7164b8cf9e5a12789e4"}
Sep 16 12:25:55.409415 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:55.409071    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2" event={"ID":"4b05d9a5-7018-4691-b5d7-c2c274c5109d","Type":"ContainerStarted","Data":"7057ceb1f3461347f0f48c8a26a2dc2b1757646048222fadd2ee9025d8d6e3ef"}
Sep 16 12:25:55.428093 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:25:55.428049    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2" podStartSLOduration=1.428031097 podStartE2EDuration="1.428031097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:55.425006182 +0000 UTC m=+1351.560328152" watchObservedRunningTime="2026-09-16 12:25:55.428031097 +0000 UTC m=+1351.563353069"
Sep 16 12:26:04.434887 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:04.434852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:26:04.435338 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:04.434906    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:26:04.447618 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:04.447596    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:26:04.469164 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:04.469143    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:26:15.715953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.715921    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"]
Sep 16 12:26:15.720596 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.720575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:15.723353 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.723324    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-51ae50\""
Sep 16 12:26:15.723487 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.723324    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-51ae50-dockercfg-24r2w\""
Sep 16 12:26:15.730427 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.730404    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"]
Sep 16 12:26:15.773716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.773683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cac034-1725-41a5-ad7d-690671b7d6d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:15.773716 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.773719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56s4w\" (UniqueName: \"kubernetes.io/projected/e5cac034-1725-41a5-ad7d-690671b7d6d8-kube-api-access-56s4w\") pod \"payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:15.874541 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.874509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cac034-1725-41a5-ad7d-690671b7d6d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:15.874541 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.874544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56s4w\" (UniqueName: \"kubernetes.io/projected/e5cac034-1725-41a5-ad7d-690671b7d6d8-kube-api-access-56s4w\") pod \"payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:15.875269 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.875251    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cac034-1725-41a5-ad7d-690671b7d6d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:15.882994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:15.882967    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56s4w\" (UniqueName: \"kubernetes.io/projected/e5cac034-1725-41a5-ad7d-690671b7d6d8-kube-api-access-56s4w\") pod \"payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:16.033868 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:16.033799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:16.304915 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:16.304890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"]
Sep 16 12:26:16.307017 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:26:16.306989    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cac034_1725_41a5_ad7d_690671b7d6d8.slice/crio-033eb244c78e4d68b869668de876f83c718eeebb4933f2bc4edc12ab1e893c69 WatchSource:0}: Error finding container 033eb244c78e4d68b869668de876f83c718eeebb4933f2bc4edc12ab1e893c69: Status 404 returned error can't find the container with id 033eb244c78e4d68b869668de876f83c718eeebb4933f2bc4edc12ab1e893c69
Sep 16 12:26:16.528489 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:16.528455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf" event={"ID":"e5cac034-1725-41a5-ad7d-690671b7d6d8","Type":"ContainerStarted","Data":"6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77"}
Sep 16 12:26:16.528694 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:16.528531    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf" event={"ID":"e5cac034-1725-41a5-ad7d-690671b7d6d8","Type":"ContainerStarted","Data":"033eb244c78e4d68b869668de876f83c718eeebb4933f2bc4edc12ab1e893c69"}
Sep 16 12:26:16.528694 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:16.528603    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:16.545461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:16.545405    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf" podStartSLOduration=1.545391062 podStartE2EDuration="1.545391062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:16.544307846 +0000 UTC m=+1372.679629818" watchObservedRunningTime="2026-09-16 12:26:16.545391062 +0000 UTC m=+1372.680713033"
Sep 16 12:26:27.535973 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:27.535941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:26:59.592615 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.592578    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"]
Sep 16 12:26:59.596557 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.596534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:26:59.599199 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.599177    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d5891124-dockercfg-7h7zx\""
Sep 16 12:26:59.599736 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.599718    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d5891124\""
Sep 16 12:26:59.612063 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.612040    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"]
Sep 16 12:26:59.694270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.694236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18b808b-7db0-46b6-ba37-69183484b6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:26:59.694389 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.694369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff75c\" (UniqueName: \"kubernetes.io/projected/d18b808b-7db0-46b6-ba37-69183484b6d2-kube-api-access-ff75c\") pod \"payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:26:59.795004 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.794976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18b808b-7db0-46b6-ba37-69183484b6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:26:59.795125 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.795043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff75c\" (UniqueName: \"kubernetes.io/projected/d18b808b-7db0-46b6-ba37-69183484b6d2-kube-api-access-ff75c\") pod \"payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:26:59.795632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.795614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18b808b-7db0-46b6-ba37-69183484b6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:26:59.803249 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.803217    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff75c\" (UniqueName: \"kubernetes.io/projected/d18b808b-7db0-46b6-ba37-69183484b6d2-kube-api-access-ff75c\") pod \"payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:26:59.909396 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:26:59.909366    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:27:00.184039 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:00.183685    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"]
Sep 16 12:27:00.186551 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:27:00.186518    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18b808b_7db0_46b6_ba37_69183484b6d2.slice/crio-9366dd016efb096a7fdecd61e232d194c8ca7134eb96c4302a70f37bd0e0281f WatchSource:0}: Error finding container 9366dd016efb096a7fdecd61e232d194c8ca7134eb96c4302a70f37bd0e0281f: Status 404 returned error can't find the container with id 9366dd016efb096a7fdecd61e232d194c8ca7134eb96c4302a70f37bd0e0281f
Sep 16 12:27:00.798207 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:00.798170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp" event={"ID":"d18b808b-7db0-46b6-ba37-69183484b6d2","Type":"ContainerStarted","Data":"6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e"}
Sep 16 12:27:00.798570 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:00.798230    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:27:00.798570 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:00.798250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp" event={"ID":"d18b808b-7db0-46b6-ba37-69183484b6d2","Type":"ContainerStarted","Data":"9366dd016efb096a7fdecd61e232d194c8ca7134eb96c4302a70f37bd0e0281f"}
Sep 16 12:27:00.815912 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:00.815856    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp" podStartSLOduration=1.815836099 podStartE2EDuration="1.815836099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:00.81301641 +0000 UTC m=+1416.948338384" watchObservedRunningTime="2026-09-16 12:27:00.815836099 +0000 UTC m=+1416.951158072"
Sep 16 12:27:06.101441 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.101411    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"]
Sep 16 12:27:06.105263 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.105245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"
Sep 16 12:27:06.107949 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.107928    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:27:06.108224 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.108209    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:27:06.112391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.112354    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"]
Sep 16 12:27:06.150780 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.150756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15318bec-e509-4d08-bb97-4ce70e737a03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd\" (UID: \"15318bec-e509-4d08-bb97-4ce70e737a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"
Sep 16 12:27:06.251417 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.251395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15318bec-e509-4d08-bb97-4ce70e737a03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd\" (UID: \"15318bec-e509-4d08-bb97-4ce70e737a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"
Sep 16 12:27:06.251993 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.251973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15318bec-e509-4d08-bb97-4ce70e737a03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd\" (UID: \"15318bec-e509-4d08-bb97-4ce70e737a03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"
Sep 16 12:27:06.418593 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.418529    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"
Sep 16 12:27:06.688849 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.688825    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"]
Sep 16 12:27:06.690860 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:27:06.690833    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15318bec_e509_4d08_bb97_4ce70e737a03.slice/crio-ee7b71cfd93a1e3ba947bfa788c2b7a2f9d320c7729db8f47245d90825975ec2 WatchSource:0}: Error finding container ee7b71cfd93a1e3ba947bfa788c2b7a2f9d320c7729db8f47245d90825975ec2: Status 404 returned error can't find the container with id ee7b71cfd93a1e3ba947bfa788c2b7a2f9d320c7729db8f47245d90825975ec2
Sep 16 12:27:06.833143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:06.833113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd" event={"ID":"15318bec-e509-4d08-bb97-4ce70e737a03","Type":"ContainerStarted","Data":"ee7b71cfd93a1e3ba947bfa788c2b7a2f9d320c7729db8f47245d90825975ec2"}
Sep 16 12:27:07.840724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:07.840695    2560 generic.go:356] "Generic (PLEG): container finished" podID="15318bec-e509-4d08-bb97-4ce70e737a03" containerID="9fbae2795e397b4d420b7627e9473d48f294a4c57388fc8f5163c17c3b71d83e" exitCode=0
Sep 16 12:27:07.841098 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:07.840729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd" event={"ID":"15318bec-e509-4d08-bb97-4ce70e737a03","Type":"ContainerDied","Data":"9fbae2795e397b4d420b7627e9473d48f294a4c57388fc8f5163c17c3b71d83e"}
Sep 16 12:27:08.998266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:08.998242    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"
Sep 16 12:27:09.076242 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:09.076215    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15318bec-e509-4d08-bb97-4ce70e737a03-maas-api-service-ca\") pod \"15318bec-e509-4d08-bb97-4ce70e737a03\" (UID: \"15318bec-e509-4d08-bb97-4ce70e737a03\") "
Sep 16 12:27:09.076509 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:09.076479    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15318bec-e509-4d08-bb97-4ce70e737a03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15318bec-e509-4d08-bb97-4ce70e737a03" (UID: "15318bec-e509-4d08-bb97-4ce70e737a03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:09.177349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:09.177292    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15318bec-e509-4d08-bb97-4ce70e737a03-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:09.860850 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:09.860823    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"
Sep 16 12:27:09.860850 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:09.860839    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd" event={"ID":"15318bec-e509-4d08-bb97-4ce70e737a03","Type":"ContainerDied","Data":"ee7b71cfd93a1e3ba947bfa788c2b7a2f9d320c7729db8f47245d90825975ec2"}
Sep 16 12:27:09.861096 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:09.860889    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee7b71cfd93a1e3ba947bfa788c2b7a2f9d320c7729db8f47245d90825975ec2"
Sep 16 12:27:11.806042 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:11.806012    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:27:18.872974 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:18.872773    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"]
Sep 16 12:27:18.873367 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:18.873025    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf" podUID="a487f889-d060-4b25-8909-93325abee717" containerName="payload-pre-processing" containerID="cri-o://83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5" gracePeriod=30
Sep 16 12:27:19.030397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.030343    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"]
Sep 16 12:27:19.032469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.032434    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f79e22-p5vqd"]
Sep 16 12:27:19.143440 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.143416    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:27:19.271082 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.271057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twp75\" (UniqueName: \"kubernetes.io/projected/a487f889-d060-4b25-8909-93325abee717-kube-api-access-twp75\") pod \"a487f889-d060-4b25-8909-93325abee717\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") "
Sep 16 12:27:19.271217 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.271152    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a487f889-d060-4b25-8909-93325abee717-plugins-config-volume\") pod \"a487f889-d060-4b25-8909-93325abee717\" (UID: \"a487f889-d060-4b25-8909-93325abee717\") "
Sep 16 12:27:19.271506 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.271486    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a487f889-d060-4b25-8909-93325abee717-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a487f889-d060-4b25-8909-93325abee717" (UID: "a487f889-d060-4b25-8909-93325abee717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:19.273283 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.273263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a487f889-d060-4b25-8909-93325abee717-kube-api-access-twp75" (OuterVolumeSpecName: "kube-api-access-twp75") pod "a487f889-d060-4b25-8909-93325abee717" (UID: "a487f889-d060-4b25-8909-93325abee717"). InnerVolumeSpecName "kube-api-access-twp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:19.372013 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.371988    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a487f889-d060-4b25-8909-93325abee717-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.372013 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.372012    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twp75\" (UniqueName: \"kubernetes.io/projected/a487f889-d060-4b25-8909-93325abee717-kube-api-access-twp75\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:19.919170 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.919139    2560 generic.go:356] "Generic (PLEG): container finished" podID="a487f889-d060-4b25-8909-93325abee717" containerID="83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5" exitCode=0
Sep 16 12:27:19.919596 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.919181    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"
Sep 16 12:27:19.919596 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.919188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf" event={"ID":"a487f889-d060-4b25-8909-93325abee717","Type":"ContainerDied","Data":"83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5"}
Sep 16 12:27:19.919596 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.919243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf" event={"ID":"a487f889-d060-4b25-8909-93325abee717","Type":"ContainerDied","Data":"a73dd74bb6090f074804a568ba26a5127f1186ef1b012b0252ecfcac17f4c2f7"}
Sep 16 12:27:19.919596 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.919266    2560 scope.go:117] "RemoveContainer" containerID="83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5"
Sep 16 12:27:19.931214 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.931196    2560 scope.go:117] "RemoveContainer" containerID="83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5"
Sep 16 12:27:19.931504 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:27:19.931482    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5\": container with ID starting with 83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5 not found: ID does not exist" containerID="83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5"
Sep 16 12:27:19.931573 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.931511    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5"} err="failed to get container status \"83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5\": rpc error: code = NotFound desc = could not find container \"83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5\": container with ID starting with 83e9dcc64e9d34818c6662833c661124b0cf665cbe0540309f55531de71708a5 not found: ID does not exist"
Sep 16 12:27:19.943047 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.943025    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"]
Sep 16 12:27:19.945938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:19.945917    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f79e22-65c94f966-k9trf"]
Sep 16 12:27:20.326194 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:20.326117    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15318bec-e509-4d08-bb97-4ce70e737a03" path="/var/lib/kubelet/pods/15318bec-e509-4d08-bb97-4ce70e737a03/volumes"
Sep 16 12:27:20.326468 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:20.326455    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a487f889-d060-4b25-8909-93325abee717" path="/var/lib/kubelet/pods/a487f889-d060-4b25-8909-93325abee717/volumes"
Sep 16 12:27:24.429172 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:24.429144    2560 scope.go:117] "RemoveContainer" containerID="9020e46df2184ba7458a907eb4169bae170aac460e02f3c5c6b603872f1954fc"
Sep 16 12:27:26.094332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.094300    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"]
Sep 16 12:27:26.094724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.094711    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a487f889-d060-4b25-8909-93325abee717" containerName="payload-pre-processing"
Sep 16 12:27:26.094768 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.094726    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a487f889-d060-4b25-8909-93325abee717" containerName="payload-pre-processing"
Sep 16 12:27:26.094768 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.094742    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15318bec-e509-4d08-bb97-4ce70e737a03" containerName="revoke-keys"
Sep 16 12:27:26.094768 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.094748    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="15318bec-e509-4d08-bb97-4ce70e737a03" containerName="revoke-keys"
Sep 16 12:27:26.094889 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.094831    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="15318bec-e509-4d08-bb97-4ce70e737a03" containerName="revoke-keys"
Sep 16 12:27:26.094889 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.094841    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a487f889-d060-4b25-8909-93325abee717" containerName="payload-pre-processing"
Sep 16 12:27:26.099265 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.099250    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"
Sep 16 12:27:26.102981 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.102866    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:27:26.103730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.102874    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:27:26.104699 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.104677    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"]
Sep 16 12:27:26.236483 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.236452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f794686-0a28-40fa-9173-db37ef7d7b6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4\" (UID: \"2f794686-0a28-40fa-9173-db37ef7d7b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"
Sep 16 12:27:26.337891 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.337862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f794686-0a28-40fa-9173-db37ef7d7b6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4\" (UID: \"2f794686-0a28-40fa-9173-db37ef7d7b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"
Sep 16 12:27:26.338403 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.338384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f794686-0a28-40fa-9173-db37ef7d7b6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4\" (UID: \"2f794686-0a28-40fa-9173-db37ef7d7b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"
Sep 16 12:27:26.415214 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.415152    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"
Sep 16 12:27:26.686125 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.686097    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"]
Sep 16 12:27:26.688156 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:27:26.688121    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f794686_0a28_40fa_9173_db37ef7d7b6b.slice/crio-308600e3735b6df2177d8ce6d62a5e4fdce53dce1a468d73f619a8db80c94c3c WatchSource:0}: Error finding container 308600e3735b6df2177d8ce6d62a5e4fdce53dce1a468d73f619a8db80c94c3c: Status 404 returned error can't find the container with id 308600e3735b6df2177d8ce6d62a5e4fdce53dce1a468d73f619a8db80c94c3c
Sep 16 12:27:26.961750 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.961723    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4" event={"ID":"2f794686-0a28-40fa-9173-db37ef7d7b6b","Type":"ContainerStarted","Data":"d93ae451994d76878f73ed321e795c1dfd8353052d1e6c95fc8b4eaf0c1fc327"}
Sep 16 12:27:26.961874 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.961777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4" event={"ID":"2f794686-0a28-40fa-9173-db37ef7d7b6b","Type":"ContainerStarted","Data":"308600e3735b6df2177d8ce6d62a5e4fdce53dce1a468d73f619a8db80c94c3c"}
Sep 16 12:27:26.976355 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:26.976309    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4" podStartSLOduration=0.976294916 podStartE2EDuration="976.294916ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:26.975196966 +0000 UTC m=+1443.110518963" watchObservedRunningTime="2026-09-16 12:27:26.976294916 +0000 UTC m=+1443.111616924"
Sep 16 12:27:27.968861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:27.968826    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f794686-0a28-40fa-9173-db37ef7d7b6b" containerID="d93ae451994d76878f73ed321e795c1dfd8353052d1e6c95fc8b4eaf0c1fc327" exitCode=0
Sep 16 12:27:27.969273 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:27.968900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4" event={"ID":"2f794686-0a28-40fa-9173-db37ef7d7b6b","Type":"ContainerDied","Data":"d93ae451994d76878f73ed321e795c1dfd8353052d1e6c95fc8b4eaf0c1fc327"}
Sep 16 12:27:29.121041 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:29.121010    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"
Sep 16 12:27:29.267096 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:29.267021    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f794686-0a28-40fa-9173-db37ef7d7b6b-maas-api-service-ca\") pod \"2f794686-0a28-40fa-9173-db37ef7d7b6b\" (UID: \"2f794686-0a28-40fa-9173-db37ef7d7b6b\") "
Sep 16 12:27:29.267366 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:29.267346    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f794686-0a28-40fa-9173-db37ef7d7b6b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f794686-0a28-40fa-9173-db37ef7d7b6b" (UID: "2f794686-0a28-40fa-9173-db37ef7d7b6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:29.367972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:29.367946    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f794686-0a28-40fa-9173-db37ef7d7b6b-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:29.987001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:29.986959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4" event={"ID":"2f794686-0a28-40fa-9173-db37ef7d7b6b","Type":"ContainerDied","Data":"308600e3735b6df2177d8ce6d62a5e4fdce53dce1a468d73f619a8db80c94c3c"}
Sep 16 12:27:29.987001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:29.986978    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"
Sep 16 12:27:29.987288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:29.987029    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="308600e3735b6df2177d8ce6d62a5e4fdce53dce1a468d73f619a8db80c94c3c"
Sep 16 12:27:36.690573 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.690538    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"]
Sep 16 12:27:36.691123 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.690844    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw" podUID="2c928568-e9da-4add-bebb-43cf7bc5208e" containerName="payload-processing" containerID="cri-o://f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d" gracePeriod=30
Sep 16 12:27:36.701113 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.701078    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"]
Sep 16 12:27:36.702346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.702269    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4" podUID="d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" containerName="payload-pre-processing" containerID="cri-o://fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8" gracePeriod=30
Sep 16 12:27:36.841326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.841281    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"]
Sep 16 12:27:36.844108 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.844076    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef3317-rfvm4"]
Sep 16 12:27:36.985339 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.985315    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:27:36.988646 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:36.988627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:27:37.031511 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.031477    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c928568-e9da-4add-bebb-43cf7bc5208e" containerID="f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d" exitCode=0
Sep 16 12:27:37.031511 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.031506    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"
Sep 16 12:27:37.031692 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.031549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw" event={"ID":"2c928568-e9da-4add-bebb-43cf7bc5208e","Type":"ContainerDied","Data":"f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d"}
Sep 16 12:27:37.031692 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.031596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw" event={"ID":"2c928568-e9da-4add-bebb-43cf7bc5208e","Type":"ContainerDied","Data":"2c1e9341c2aac97dec8a60cf1ae41f9af17c76fe015d0dda162e93b7be965356"}
Sep 16 12:27:37.031692 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.031625    2560 scope.go:117] "RemoveContainer" containerID="f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d"
Sep 16 12:27:37.033281 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.033257    2560 generic.go:356] "Generic (PLEG): container finished" podID="d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" containerID="fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8" exitCode=0
Sep 16 12:27:37.033410 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.033296    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"
Sep 16 12:27:37.033410 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.033314    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4" event={"ID":"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c","Type":"ContainerDied","Data":"fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8"}
Sep 16 12:27:37.033410 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.033356    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4" event={"ID":"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c","Type":"ContainerDied","Data":"4a04c1e7b0070e6705feb6963652d2592219a633be18ce1ba8b3219209b0cadb"}
Sep 16 12:27:37.040572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.040555    2560 scope.go:117] "RemoveContainer" containerID="f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d"
Sep 16 12:27:37.040875 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:27:37.040851    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d\": container with ID starting with f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d not found: ID does not exist" containerID="f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d"
Sep 16 12:27:37.041001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.040881    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d"} err="failed to get container status \"f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d\": rpc error: code = NotFound desc = could not find container \"f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d\": container with ID starting with f98bf7667a0c7710a3860d8a0cda9dc237120b30700e70c7087172d734f83d9d not found: ID does not exist"
Sep 16 12:27:37.041001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.040897    2560 scope.go:117] "RemoveContainer" containerID="fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8"
Sep 16 12:27:37.048948 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.048928    2560 scope.go:117] "RemoveContainer" containerID="fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8"
Sep 16 12:27:37.049217 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:27:37.049199    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8\": container with ID starting with fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8 not found: ID does not exist" containerID="fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8"
Sep 16 12:27:37.049293 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.049222    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8"} err="failed to get container status \"fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8\": rpc error: code = NotFound desc = could not find container \"fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8\": container with ID starting with fd88114a052ea0bbf571cae7363c41a925c9006ab1fa65be72d85d29193942a8 not found: ID does not exist"
Sep 16 12:27:37.138467 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.138444    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvc22\" (UniqueName: \"kubernetes.io/projected/2c928568-e9da-4add-bebb-43cf7bc5208e-kube-api-access-nvc22\") pod \"2c928568-e9da-4add-bebb-43cf7bc5208e\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") "
Sep 16 12:27:37.138554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.138500    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c928568-e9da-4add-bebb-43cf7bc5208e-plugins-config-volume\") pod \"2c928568-e9da-4add-bebb-43cf7bc5208e\" (UID: \"2c928568-e9da-4add-bebb-43cf7bc5208e\") "
Sep 16 12:27:37.138554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.138532    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvzf2\" (UniqueName: \"kubernetes.io/projected/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-kube-api-access-bvzf2\") pod \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") "
Sep 16 12:27:37.138643 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.138569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-plugins-config-volume\") pod \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\" (UID: \"d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c\") "
Sep 16 12:27:37.138959 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.138935    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" (UID: "d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:37.138959 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.138942    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c928568-e9da-4add-bebb-43cf7bc5208e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c928568-e9da-4add-bebb-43cf7bc5208e" (UID: "2c928568-e9da-4add-bebb-43cf7bc5208e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:37.140562 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.140546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c928568-e9da-4add-bebb-43cf7bc5208e-kube-api-access-nvc22" (OuterVolumeSpecName: "kube-api-access-nvc22") pod "2c928568-e9da-4add-bebb-43cf7bc5208e" (UID: "2c928568-e9da-4add-bebb-43cf7bc5208e"). InnerVolumeSpecName "kube-api-access-nvc22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:37.140662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.140643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-kube-api-access-bvzf2" (OuterVolumeSpecName: "kube-api-access-bvzf2") pod "d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" (UID: "d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c"). InnerVolumeSpecName "kube-api-access-bvzf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:37.239662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.239611    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvc22\" (UniqueName: \"kubernetes.io/projected/2c928568-e9da-4add-bebb-43cf7bc5208e-kube-api-access-nvc22\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.239662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.239635    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c928568-e9da-4add-bebb-43cf7bc5208e-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.239662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.239644    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvzf2\" (UniqueName: \"kubernetes.io/projected/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-kube-api-access-bvzf2\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.239662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.239652    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:27:37.355560 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.355528    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"]
Sep 16 12:27:37.358101 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.358063    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef3317-6c548d6886-c42m4"]
Sep 16 12:27:37.368194 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.368174    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"]
Sep 16 12:27:37.371035 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:37.371013    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef3317-d77796fb5-lbxvw"]
Sep 16 12:27:38.326764 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:38.326731    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c928568-e9da-4add-bebb-43cf7bc5208e" path="/var/lib/kubelet/pods/2c928568-e9da-4add-bebb-43cf7bc5208e/volumes"
Sep 16 12:27:38.327132 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:38.327075    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f794686-0a28-40fa-9173-db37ef7d7b6b" path="/var/lib/kubelet/pods/2f794686-0a28-40fa-9173-db37ef7d7b6b/volumes"
Sep 16 12:27:38.327397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:38.327383    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" path="/var/lib/kubelet/pods/d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c/volumes"
Sep 16 12:27:52.438801 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.438746    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"]
Sep 16 12:27:52.439421 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439401    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" containerName="payload-pre-processing"
Sep 16 12:27:52.439469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439425    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" containerName="payload-pre-processing"
Sep 16 12:27:52.439469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439453    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c928568-e9da-4add-bebb-43cf7bc5208e" containerName="payload-processing"
Sep 16 12:27:52.439469 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439462    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c928568-e9da-4add-bebb-43cf7bc5208e" containerName="payload-processing"
Sep 16 12:27:52.439563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439474    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f794686-0a28-40fa-9173-db37ef7d7b6b" containerName="revoke-keys"
Sep 16 12:27:52.439563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439483    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f794686-0a28-40fa-9173-db37ef7d7b6b" containerName="revoke-keys"
Sep 16 12:27:52.439625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439576    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f794686-0a28-40fa-9173-db37ef7d7b6b" containerName="revoke-keys"
Sep 16 12:27:52.439625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439589    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c21f44-34b8-4ac5-9cfa-f8aa73909b1c" containerName="payload-pre-processing"
Sep 16 12:27:52.439625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.439599    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c928568-e9da-4add-bebb-43cf7bc5208e" containerName="payload-processing"
Sep 16 12:27:52.445167 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.445146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.449397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.449369    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"]
Sep 16 12:27:52.449589 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.449428    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d5891124\"/\"e2e-lifecycle-model-d5891124-kserve-self-signed-certs\""
Sep 16 12:27:52.449718 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.449600    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d5891124\"/\"openshift-service-ca.crt\""
Sep 16 12:27:52.449718 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.449571    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d5891124\"/\"default-dockercfg-27zvm\""
Sep 16 12:27:52.449718 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.449535    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d5891124\"/\"kube-root-ca.crt\""
Sep 16 12:27:52.477733 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.477707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-dshm\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.477868 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.477759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-tmp-dir\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.477868 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.477841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmv4w\" (UniqueName: \"kubernetes.io/projected/a7e80238-0346-4063-9d9f-b632f8eb4838-kube-api-access-kmv4w\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.477948 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.477921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e80238-0346-4063-9d9f-b632f8eb4838-tls-certs\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.477983 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.477964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-home\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.478019 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.477992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-model-cache\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579036 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-dshm\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579154 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-tmp-dir\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579154 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmv4w\" (UniqueName: \"kubernetes.io/projected/a7e80238-0346-4063-9d9f-b632f8eb4838-kube-api-access-kmv4w\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579154 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e80238-0346-4063-9d9f-b632f8eb4838-tls-certs\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579154 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-home\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579362 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-model-cache\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-tmp-dir\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579608 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-model-cache\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.579608 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.579580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-home\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.581419 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.581402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-dshm\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.581671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.581654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e80238-0346-4063-9d9f-b632f8eb4838-tls-certs\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.586597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.586574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmv4w\" (UniqueName: \"kubernetes.io/projected/a7e80238-0346-4063-9d9f-b632f8eb4838-kube-api-access-kmv4w\") pod \"e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") " pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:52.757307 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:52.757230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:27:53.027865 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:53.027841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"]
Sep 16 12:27:53.029563 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:27:53.029535    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7e80238_0346_4063_9d9f_b632f8eb4838.slice/crio-80e7f7372eb8db4a10402a1b76dea2d3f692bb8f47eaecb46a748b474eeb61b9 WatchSource:0}: Error finding container 80e7f7372eb8db4a10402a1b76dea2d3f692bb8f47eaecb46a748b474eeb61b9: Status 404 returned error can't find the container with id 80e7f7372eb8db4a10402a1b76dea2d3f692bb8f47eaecb46a748b474eeb61b9
Sep 16 12:27:53.123497 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:53.123462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv" event={"ID":"a7e80238-0346-4063-9d9f-b632f8eb4838","Type":"ContainerStarted","Data":"8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3"}
Sep 16 12:27:53.123622 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:53.123509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv" event={"ID":"a7e80238-0346-4063-9d9f-b632f8eb4838","Type":"ContainerStarted","Data":"80e7f7372eb8db4a10402a1b76dea2d3f692bb8f47eaecb46a748b474eeb61b9"}
Sep 16 12:27:53.142027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:27:53.141983    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv" podStartSLOduration=1.1419674930000001 podStartE2EDuration="1.141967493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:53.140306958 +0000 UTC m=+1469.275628933" watchObservedRunningTime="2026-09-16 12:27:53.141967493 +0000 UTC m=+1469.277289530"
Sep 16 12:28:02.757705 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:02.757669    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:28:02.758117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:02.757729    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:28:02.770137 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:02.770117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:28:03.199140 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:03.199107    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:28:24.383086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:24.383055    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:28:24.384977 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:24.384958    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:28:24.471898 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:24.471865    2560 scope.go:117] "RemoveContainer" containerID="4ae3266b53f07ed15c39d0b7969bcfabc6f5ae6e2d518526ce35cd881d4742e0"
Sep 16 12:28:24.481988 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:24.481963    2560 scope.go:117] "RemoveContainer" containerID="9694429967c62a717e9febc3adab949969c277aabfc76512a435a9cb782ee995"
Sep 16 12:28:27.545070 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.545033    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"]
Sep 16 12:28:27.550326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.550305    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.554246 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.554222    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0a9008\""
Sep 16 12:28:27.554388 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.554329    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0a9008-dockercfg-cdpt2\""
Sep 16 12:28:27.561282 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.561259    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"]
Sep 16 12:28:27.582813 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.582778    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"]
Sep 16 12:28:27.586317 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.586301    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:27.596939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.596917    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"]
Sep 16 12:28:27.614230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.614197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f070e4b1-f730-4fbc-bbff-7c490ab7f230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:27.614358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.614250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4vc\" (UniqueName: \"kubernetes.io/projected/f070e4b1-f730-4fbc-bbff-7c490ab7f230-kube-api-access-wr4vc\") pod \"payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:27.614358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.614346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.614476 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.614381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtng\" (UniqueName: \"kubernetes.io/projected/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-kube-api-access-mxtng\") pod \"payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.714981 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.714949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtng\" (UniqueName: \"kubernetes.io/projected/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-kube-api-access-mxtng\") pod \"payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.715152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.715064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f070e4b1-f730-4fbc-bbff-7c490ab7f230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:27.715152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.715083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4vc\" (UniqueName: \"kubernetes.io/projected/f070e4b1-f730-4fbc-bbff-7c490ab7f230-kube-api-access-wr4vc\") pod \"payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:27.715152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.715148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.716020 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.715997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.716082 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.716005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f070e4b1-f730-4fbc-bbff-7c490ab7f230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:27.724183 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.724154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4vc\" (UniqueName: \"kubernetes.io/projected/f070e4b1-f730-4fbc-bbff-7c490ab7f230-kube-api-access-wr4vc\") pod \"payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:27.724295 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.724273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtng\" (UniqueName: \"kubernetes.io/projected/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-kube-api-access-mxtng\") pod \"payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.862911 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.862883    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:27.898023 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:27.897994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:28.145001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.144920    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"]
Sep 16 12:28:28.147991 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:28:28.147967    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6357ed4_8477_4a9f_85d5_da8d3c0fab48.slice/crio-f8cb93c03479b5f88f7b321e8d42a3d31442501c0811c9b1a226cd314cb09ec2 WatchSource:0}: Error finding container f8cb93c03479b5f88f7b321e8d42a3d31442501c0811c9b1a226cd314cb09ec2: Status 404 returned error can't find the container with id f8cb93c03479b5f88f7b321e8d42a3d31442501c0811c9b1a226cd314cb09ec2
Sep 16 12:28:28.182118 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.182095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"]
Sep 16 12:28:28.184744 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:28:28.184720    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf070e4b1_f730_4fbc_bbff_7c490ab7f230.slice/crio-26a3dd7f9d00f2a5d3eca80666e5df071aaf56b8fce1b5dd24784f1f6f6d3975 WatchSource:0}: Error finding container 26a3dd7f9d00f2a5d3eca80666e5df071aaf56b8fce1b5dd24784f1f6f6d3975: Status 404 returned error can't find the container with id 26a3dd7f9d00f2a5d3eca80666e5df071aaf56b8fce1b5dd24784f1f6f6d3975
Sep 16 12:28:28.336708 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.336673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7" event={"ID":"e6357ed4-8477-4a9f-85d5-da8d3c0fab48","Type":"ContainerStarted","Data":"4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423"}
Sep 16 12:28:28.336929 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.336737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7" event={"ID":"e6357ed4-8477-4a9f-85d5-da8d3c0fab48","Type":"ContainerStarted","Data":"f8cb93c03479b5f88f7b321e8d42a3d31442501c0811c9b1a226cd314cb09ec2"}
Sep 16 12:28:28.336929 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.336811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:28.338480 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.338456    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg" event={"ID":"f070e4b1-f730-4fbc-bbff-7c490ab7f230","Type":"ContainerStarted","Data":"986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8"}
Sep 16 12:28:28.338582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.338502    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:28.338582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.338521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg" event={"ID":"f070e4b1-f730-4fbc-bbff-7c490ab7f230","Type":"ContainerStarted","Data":"26a3dd7f9d00f2a5d3eca80666e5df071aaf56b8fce1b5dd24784f1f6f6d3975"}
Sep 16 12:28:28.350820 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.350756    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7" podStartSLOduration=1.35073756 podStartE2EDuration="1.35073756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:28.350263915 +0000 UTC m=+1504.485585887" watchObservedRunningTime="2026-09-16 12:28:28.35073756 +0000 UTC m=+1504.486059534"
Sep 16 12:28:28.365416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:28.365373    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg" podStartSLOduration=1.365359642 podStartE2EDuration="1.365359642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:28.364008334 +0000 UTC m=+1504.499330305" watchObservedRunningTime="2026-09-16 12:28:28.365359642 +0000 UTC m=+1504.500681613"
Sep 16 12:28:34.581170 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.581131    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"]
Sep 16 12:28:34.584795 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.584764    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"
Sep 16 12:28:34.587461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.587442    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:28:34.587561 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.587485    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:28:34.591594 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.591571    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"]
Sep 16 12:28:34.674822 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.674795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43bdfb2-9def-4d35-8388-606d04e8ec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5891124-wxrh2\" (UID: \"f43bdfb2-9def-4d35-8388-606d04e8ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"
Sep 16 12:28:34.775297 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.775272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43bdfb2-9def-4d35-8388-606d04e8ec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5891124-wxrh2\" (UID: \"f43bdfb2-9def-4d35-8388-606d04e8ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"
Sep 16 12:28:34.775862 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.775846    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43bdfb2-9def-4d35-8388-606d04e8ec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5891124-wxrh2\" (UID: \"f43bdfb2-9def-4d35-8388-606d04e8ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"
Sep 16 12:28:34.898493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:34.898471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"
Sep 16 12:28:35.170952 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:35.170929    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"]
Sep 16 12:28:35.173167 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:28:35.173138    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43bdfb2_9def_4d35_8388_606d04e8ec77.slice/crio-9ed3d127278a8569daad81f753b5069655f3e44dab062ee164040a0c8e1bad2a WatchSource:0}: Error finding container 9ed3d127278a8569daad81f753b5069655f3e44dab062ee164040a0c8e1bad2a: Status 404 returned error can't find the container with id 9ed3d127278a8569daad81f753b5069655f3e44dab062ee164040a0c8e1bad2a
Sep 16 12:28:35.379917 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:35.379887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2" event={"ID":"f43bdfb2-9def-4d35-8388-606d04e8ec77","Type":"ContainerStarted","Data":"9ed3d127278a8569daad81f753b5069655f3e44dab062ee164040a0c8e1bad2a"}
Sep 16 12:28:36.387383 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:36.387347    2560 generic.go:356] "Generic (PLEG): container finished" podID="f43bdfb2-9def-4d35-8388-606d04e8ec77" containerID="05f0488ad9b0e4aacdcb7a813177477608763eab35e64b52b841af922e89269b" exitCode=0
Sep 16 12:28:36.387724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:36.387427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2" event={"ID":"f43bdfb2-9def-4d35-8388-606d04e8ec77","Type":"ContainerDied","Data":"05f0488ad9b0e4aacdcb7a813177477608763eab35e64b52b841af922e89269b"}
Sep 16 12:28:37.533946 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:37.533926    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"
Sep 16 12:28:37.600866 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:37.600840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43bdfb2-9def-4d35-8388-606d04e8ec77-maas-api-service-ca\") pod \"f43bdfb2-9def-4d35-8388-606d04e8ec77\" (UID: \"f43bdfb2-9def-4d35-8388-606d04e8ec77\") "
Sep 16 12:28:37.601224 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:37.601200    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43bdfb2-9def-4d35-8388-606d04e8ec77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f43bdfb2-9def-4d35-8388-606d04e8ec77" (UID: "f43bdfb2-9def-4d35-8388-606d04e8ec77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:37.601287 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:37.601272    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43bdfb2-9def-4d35-8388-606d04e8ec77-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:38.399722 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:38.399695    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"
Sep 16 12:28:38.399908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:38.399721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2" event={"ID":"f43bdfb2-9def-4d35-8388-606d04e8ec77","Type":"ContainerDied","Data":"9ed3d127278a8569daad81f753b5069655f3e44dab062ee164040a0c8e1bad2a"}
Sep 16 12:28:38.399908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:38.399759    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed3d127278a8569daad81f753b5069655f3e44dab062ee164040a0c8e1bad2a"
Sep 16 12:28:39.347017 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:39.346986    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:28:39.347474 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:39.347378    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:28:41.319240 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:41.319198    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"]
Sep 16 12:28:46.790138 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.790054    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"]
Sep 16 12:28:46.790522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.790508    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43bdfb2-9def-4d35-8388-606d04e8ec77" containerName="revoke-keys"
Sep 16 12:28:46.790576 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.790525    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43bdfb2-9def-4d35-8388-606d04e8ec77" containerName="revoke-keys"
Sep 16 12:28:46.790626 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.790617    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43bdfb2-9def-4d35-8388-606d04e8ec77" containerName="revoke-keys"
Sep 16 12:28:46.794030 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.794010    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:46.796654 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.796634    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0e94e5\""
Sep 16 12:28:46.803055 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.803033    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"]
Sep 16 12:28:46.886131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.886099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzrr\" (UniqueName: \"kubernetes.io/projected/9947af79-2b99-44d7-930c-ca4cf744f706-kube-api-access-ftzrr\") pod \"payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:46.886131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.886130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9947af79-2b99-44d7-930c-ca4cf744f706-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:46.986981 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.986950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzrr\" (UniqueName: \"kubernetes.io/projected/9947af79-2b99-44d7-930c-ca4cf744f706-kube-api-access-ftzrr\") pod \"payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:46.986981 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.986983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9947af79-2b99-44d7-930c-ca4cf744f706-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:46.987650 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.987632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9947af79-2b99-44d7-930c-ca4cf744f706-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:46.995054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:46.995036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzrr\" (UniqueName: \"kubernetes.io/projected/9947af79-2b99-44d7-930c-ca4cf744f706-kube-api-access-ftzrr\") pod \"payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:47.105245 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:47.105211    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:47.388641 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:47.388600    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"]
Sep 16 12:28:47.390690 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:28:47.390665    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9947af79_2b99_44d7_930c_ca4cf744f706.slice/crio-cc1179eb9b333ddfee46a4fdbf3a32d595deeb5ce0325f318779325e74eec61b WatchSource:0}: Error finding container cc1179eb9b333ddfee46a4fdbf3a32d595deeb5ce0325f318779325e74eec61b: Status 404 returned error can't find the container with id cc1179eb9b333ddfee46a4fdbf3a32d595deeb5ce0325f318779325e74eec61b
Sep 16 12:28:47.457244 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:47.457218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk" event={"ID":"9947af79-2b99-44d7-930c-ca4cf744f706","Type":"ContainerStarted","Data":"cc1179eb9b333ddfee46a4fdbf3a32d595deeb5ce0325f318779325e74eec61b"}
Sep 16 12:28:48.464126 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:48.464088    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk" event={"ID":"9947af79-2b99-44d7-930c-ca4cf744f706","Type":"ContainerStarted","Data":"1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53"}
Sep 16 12:28:48.464485 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:48.464235    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:28:48.481129 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:48.481078    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk" podStartSLOduration=2.481060178 podStartE2EDuration="2.481060178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:48.48050221 +0000 UTC m=+1524.615824183" watchObservedRunningTime="2026-09-16 12:28:48.481060178 +0000 UTC m=+1524.616382150"
Sep 16 12:28:50.501348 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.501316    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"]
Sep 16 12:28:50.501728 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.501559    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp" podUID="d18b808b-7db0-46b6-ba37-69183484b6d2" containerName="payload-pre-processing" containerID="cri-o://6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e" gracePeriod=30
Sep 16 12:28:50.705266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.705235    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"]
Sep 16 12:28:50.708893 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.708864    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5891124-wxrh2"]
Sep 16 12:28:50.769774 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.769749    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:28:50.924482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.924458    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff75c\" (UniqueName: \"kubernetes.io/projected/d18b808b-7db0-46b6-ba37-69183484b6d2-kube-api-access-ff75c\") pod \"d18b808b-7db0-46b6-ba37-69183484b6d2\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") "
Sep 16 12:28:50.924660 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.924510    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18b808b-7db0-46b6-ba37-69183484b6d2-plugins-config-volume\") pod \"d18b808b-7db0-46b6-ba37-69183484b6d2\" (UID: \"d18b808b-7db0-46b6-ba37-69183484b6d2\") "
Sep 16 12:28:50.925001 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.924971    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18b808b-7db0-46b6-ba37-69183484b6d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d18b808b-7db0-46b6-ba37-69183484b6d2" (UID: "d18b808b-7db0-46b6-ba37-69183484b6d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:50.926709 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:50.926685    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18b808b-7db0-46b6-ba37-69183484b6d2-kube-api-access-ff75c" (OuterVolumeSpecName: "kube-api-access-ff75c") pod "d18b808b-7db0-46b6-ba37-69183484b6d2" (UID: "d18b808b-7db0-46b6-ba37-69183484b6d2"). InnerVolumeSpecName "kube-api-access-ff75c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:51.025810 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.025733    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff75c\" (UniqueName: \"kubernetes.io/projected/d18b808b-7db0-46b6-ba37-69183484b6d2-kube-api-access-ff75c\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:51.025810 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.025755    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18b808b-7db0-46b6-ba37-69183484b6d2-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:51.485535 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.485496    2560 generic.go:356] "Generic (PLEG): container finished" podID="d18b808b-7db0-46b6-ba37-69183484b6d2" containerID="6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e" exitCode=0
Sep 16 12:28:51.485681 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.485611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp" event={"ID":"d18b808b-7db0-46b6-ba37-69183484b6d2","Type":"ContainerDied","Data":"6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e"}
Sep 16 12:28:51.485681 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.485672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp" event={"ID":"d18b808b-7db0-46b6-ba37-69183484b6d2","Type":"ContainerDied","Data":"9366dd016efb096a7fdecd61e232d194c8ca7134eb96c4302a70f37bd0e0281f"}
Sep 16 12:28:51.485772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.485614    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"
Sep 16 12:28:51.485772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.485705    2560 scope.go:117] "RemoveContainer" containerID="6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e"
Sep 16 12:28:51.496295 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.496268    2560 scope.go:117] "RemoveContainer" containerID="6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e"
Sep 16 12:28:51.496579 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:28:51.496554    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e\": container with ID starting with 6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e not found: ID does not exist" containerID="6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e"
Sep 16 12:28:51.496654 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.496593    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e"} err="failed to get container status \"6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e\": rpc error: code = NotFound desc = could not find container \"6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e\": container with ID starting with 6c094eb1ae1b4f58a424284199d0653dfe23896cdc9922cf040bcbccdce8d19e not found: ID does not exist"
Sep 16 12:28:51.512032 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.512010    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"]
Sep 16 12:28:51.514690 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:51.514669    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5891124-64d987f7d-zgzhp"]
Sep 16 12:28:52.326611 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:52.326573    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d18b808b-7db0-46b6-ba37-69183484b6d2" path="/var/lib/kubelet/pods/d18b808b-7db0-46b6-ba37-69183484b6d2/volumes"
Sep 16 12:28:52.327053 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:52.327031    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43bdfb2-9def-4d35-8388-606d04e8ec77" path="/var/lib/kubelet/pods/f43bdfb2-9def-4d35-8388-606d04e8ec77/volumes"
Sep 16 12:28:56.339601 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.339564    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2" podUID="4b05d9a5-7018-4691-b5d7-c2c274c5109d" containerName="main" containerID="cri-o://9cc1618ec35f5023ac9ecc2ea6df297800f40ff7f3dda7164b8cf9e5a12789e4" gracePeriod=45
Sep 16 12:28:56.519809 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.519762    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b05d9a5-7018-4691-b5d7-c2c274c5109d" containerID="9cc1618ec35f5023ac9ecc2ea6df297800f40ff7f3dda7164b8cf9e5a12789e4" exitCode=0
Sep 16 12:28:56.519965 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.519826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2" event={"ID":"4b05d9a5-7018-4691-b5d7-c2c274c5109d","Type":"ContainerDied","Data":"9cc1618ec35f5023ac9ecc2ea6df297800f40ff7f3dda7164b8cf9e5a12789e4"}
Sep 16 12:28:56.600407 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.600382    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:28:56.778583 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.778558    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tls-certs\") pod \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") "
Sep 16 12:28:56.778747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.778599    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tmp-dir\") pod \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") "
Sep 16 12:28:56.778747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.778640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5gx5\" (UniqueName: \"kubernetes.io/projected/4b05d9a5-7018-4691-b5d7-c2c274c5109d-kube-api-access-w5gx5\") pod \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") "
Sep 16 12:28:56.778747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.778659    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-home\") pod \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") "
Sep 16 12:28:56.778747 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.778738    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-dshm\") pod \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") "
Sep 16 12:28:56.779002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.778757    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-model-cache\") pod \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\" (UID: \"4b05d9a5-7018-4691-b5d7-c2c274c5109d\") "
Sep 16 12:28:56.779073 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.779013    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b05d9a5-7018-4691-b5d7-c2c274c5109d" (UID: "4b05d9a5-7018-4691-b5d7-c2c274c5109d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:56.779073 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.779032    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-home" (OuterVolumeSpecName: "home") pod "4b05d9a5-7018-4691-b5d7-c2c274c5109d" (UID: "4b05d9a5-7018-4691-b5d7-c2c274c5109d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:56.779159 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.779111    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b05d9a5-7018-4691-b5d7-c2c274c5109d" (UID: "4b05d9a5-7018-4691-b5d7-c2c274c5109d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:56.781102 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.781077    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b05d9a5-7018-4691-b5d7-c2c274c5109d" (UID: "4b05d9a5-7018-4691-b5d7-c2c274c5109d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:56.781206 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.781114    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-dshm" (OuterVolumeSpecName: "dshm") pod "4b05d9a5-7018-4691-b5d7-c2c274c5109d" (UID: "4b05d9a5-7018-4691-b5d7-c2c274c5109d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:56.781206 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.781175    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b05d9a5-7018-4691-b5d7-c2c274c5109d-kube-api-access-w5gx5" (OuterVolumeSpecName: "kube-api-access-w5gx5") pod "4b05d9a5-7018-4691-b5d7-c2c274c5109d" (UID: "4b05d9a5-7018-4691-b5d7-c2c274c5109d"). InnerVolumeSpecName "kube-api-access-w5gx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:56.879922 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.879856    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:56.879922 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.879882    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:56.879922 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.879896    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:56.879922 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.879907    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:56.879922 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.879918    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5gx5\" (UniqueName: \"kubernetes.io/projected/4b05d9a5-7018-4691-b5d7-c2c274c5109d-kube-api-access-w5gx5\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:56.880141 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:56.879931    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b05d9a5-7018-4691-b5d7-c2c274c5109d-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:28:57.527089 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:57.527062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"
Sep 16 12:28:57.527541 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:57.527056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2" event={"ID":"4b05d9a5-7018-4691-b5d7-c2c274c5109d","Type":"ContainerDied","Data":"7057ceb1f3461347f0f48c8a26a2dc2b1757646048222fadd2ee9025d8d6e3ef"}
Sep 16 12:28:57.527541 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:57.527218    2560 scope.go:117] "RemoveContainer" containerID="9cc1618ec35f5023ac9ecc2ea6df297800f40ff7f3dda7164b8cf9e5a12789e4"
Sep 16 12:28:57.550172 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:57.550146    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"]
Sep 16 12:28:57.553864 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:57.553841    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/e2e-conflict-a-786e64f4-kserve-6c5c49bcc5-5p9m2"]
Sep 16 12:28:58.324820 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:58.324770    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b05d9a5-7018-4691-b5d7-c2c274c5109d" path="/var/lib/kubelet/pods/4b05d9a5-7018-4691-b5d7-c2c274c5109d/volumes"
Sep 16 12:28:58.804041 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:58.804012    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"]
Sep 16 12:28:59.471826 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:28:59.471771    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:29:13.830238 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:13.830196    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv" podUID="a7e80238-0346-4063-9d9f-b632f8eb4838" containerName="main" containerID="cri-o://8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3" gracePeriod=45
Sep 16 12:29:14.093215 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.093189    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:29:14.133078 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e80238-0346-4063-9d9f-b632f8eb4838-tls-certs\") pod \"a7e80238-0346-4063-9d9f-b632f8eb4838\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") "
Sep 16 12:29:14.133203 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133109    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmv4w\" (UniqueName: \"kubernetes.io/projected/a7e80238-0346-4063-9d9f-b632f8eb4838-kube-api-access-kmv4w\") pod \"a7e80238-0346-4063-9d9f-b632f8eb4838\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") "
Sep 16 12:29:14.133203 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133153    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-dshm\") pod \"a7e80238-0346-4063-9d9f-b632f8eb4838\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") "
Sep 16 12:29:14.133203 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-tmp-dir\") pod \"a7e80238-0346-4063-9d9f-b632f8eb4838\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") "
Sep 16 12:29:14.133358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133263    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-model-cache\") pod \"a7e80238-0346-4063-9d9f-b632f8eb4838\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") "
Sep 16 12:29:14.133358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-home\") pod \"a7e80238-0346-4063-9d9f-b632f8eb4838\" (UID: \"a7e80238-0346-4063-9d9f-b632f8eb4838\") "
Sep 16 12:29:14.133645 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7e80238-0346-4063-9d9f-b632f8eb4838" (UID: "a7e80238-0346-4063-9d9f-b632f8eb4838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:29:14.133645 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133606    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7e80238-0346-4063-9d9f-b632f8eb4838" (UID: "a7e80238-0346-4063-9d9f-b632f8eb4838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:29:14.133834 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-home" (OuterVolumeSpecName: "home") pod "a7e80238-0346-4063-9d9f-b632f8eb4838" (UID: "a7e80238-0346-4063-9d9f-b632f8eb4838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:29:14.133834 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133802    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:29:14.133947 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.133918    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:29:14.135553 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.135519    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-dshm" (OuterVolumeSpecName: "dshm") pod "a7e80238-0346-4063-9d9f-b632f8eb4838" (UID: "a7e80238-0346-4063-9d9f-b632f8eb4838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:29:14.135766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.135744    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e80238-0346-4063-9d9f-b632f8eb4838-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7e80238-0346-4063-9d9f-b632f8eb4838" (UID: "a7e80238-0346-4063-9d9f-b632f8eb4838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:29:14.135940 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.135917    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e80238-0346-4063-9d9f-b632f8eb4838-kube-api-access-kmv4w" (OuterVolumeSpecName: "kube-api-access-kmv4w") pod "a7e80238-0346-4063-9d9f-b632f8eb4838" (UID: "a7e80238-0346-4063-9d9f-b632f8eb4838"). InnerVolumeSpecName "kube-api-access-kmv4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:29:14.234829 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.234807    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e80238-0346-4063-9d9f-b632f8eb4838-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:29:14.234829 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.234830    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmv4w\" (UniqueName: \"kubernetes.io/projected/a7e80238-0346-4063-9d9f-b632f8eb4838-kube-api-access-kmv4w\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:29:14.234971 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.234839    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:29:14.234971 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.234847    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e80238-0346-4063-9d9f-b632f8eb4838-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:29:14.623063 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.623033    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7e80238-0346-4063-9d9f-b632f8eb4838" containerID="8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3" exitCode=0
Sep 16 12:29:14.623242 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.623075    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"
Sep 16 12:29:14.623242 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.623120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv" event={"ID":"a7e80238-0346-4063-9d9f-b632f8eb4838","Type":"ContainerDied","Data":"8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3"}
Sep 16 12:29:14.623242 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.623161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv" event={"ID":"a7e80238-0346-4063-9d9f-b632f8eb4838","Type":"ContainerDied","Data":"80e7f7372eb8db4a10402a1b76dea2d3f692bb8f47eaecb46a748b474eeb61b9"}
Sep 16 12:29:14.623242 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.623178    2560 scope.go:117] "RemoveContainer" containerID="8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3"
Sep 16 12:29:14.632177 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.632158    2560 scope.go:117] "RemoveContainer" containerID="8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3"
Sep 16 12:29:14.632461 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:29:14.632438    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3\": container with ID starting with 8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3 not found: ID does not exist" containerID="8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3"
Sep 16 12:29:14.632566 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.632466    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3"} err="failed to get container status \"8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3\": rpc error: code = NotFound desc = could not find container \"8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3\": container with ID starting with 8e9c8aae47a8b5a5708727773857eac21cdece7ff4811f8aedb9e21d6ced33b3 not found: ID does not exist"
Sep 16 12:29:14.640945 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.640924    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"]
Sep 16 12:29:14.644287 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:14.644265    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d5891124/e2e-lifecycle-model-d5891124-kserve-595fc68ff5-gz8bv"]
Sep 16 12:29:16.325510 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:16.325478    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e80238-0346-4063-9d9f-b632f8eb4838" path="/var/lib/kubelet/pods/a7e80238-0346-4063-9d9f-b632f8eb4838/volumes"
Sep 16 12:29:24.519442 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:24.519405    2560 scope.go:117] "RemoveContainer" containerID="02f5824f41b2e46b33c4d32bac1651f9263f4ba130598d6b05c5f2a1c1c7d760"
Sep 16 12:29:45.247378 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247347    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"]
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247757    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d18b808b-7db0-46b6-ba37-69183484b6d2" containerName="payload-pre-processing"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247768    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d18b808b-7db0-46b6-ba37-69183484b6d2" containerName="payload-pre-processing"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247779    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e80238-0346-4063-9d9f-b632f8eb4838" containerName="main"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247803    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e80238-0346-4063-9d9f-b632f8eb4838" containerName="main"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247817    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b05d9a5-7018-4691-b5d7-c2c274c5109d" containerName="main"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247822    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b05d9a5-7018-4691-b5d7-c2c274c5109d" containerName="main"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247909    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d18b808b-7db0-46b6-ba37-69183484b6d2" containerName="payload-pre-processing"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247918    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b05d9a5-7018-4691-b5d7-c2c274c5109d" containerName="main"
Sep 16 12:29:45.249766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.247925    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e80238-0346-4063-9d9f-b632f8eb4838" containerName="main"
Sep 16 12:29:45.250941 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.250926    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.253859 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.253837    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"default-dockercfg-stkpb\""
Sep 16 12:29:45.255052 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.255034    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"kube-root-ca.crt\""
Sep 16 12:29:45.255134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.255068    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"openshift-service-ca.crt\""
Sep 16 12:29:45.255134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.255040    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"auth-test-model-0a9008-kserve-self-signed-certs\""
Sep 16 12:29:45.263388 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.263362    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"]
Sep 16 12:29:45.414414 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.414382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-model-cache\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.414572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.414449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4mn\" (UniqueName: \"kubernetes.io/projected/866d7f22-cadb-45f7-8d1d-01f3179877bd-kube-api-access-sv4mn\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.414695 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.414674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866d7f22-cadb-45f7-8d1d-01f3179877bd-tls-certs\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.414781 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.414757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-home\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.414875 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.414811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-tmp-dir\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.414945 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.414921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-dshm\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516173 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-model-cache\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516173 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4mn\" (UniqueName: \"kubernetes.io/projected/866d7f22-cadb-45f7-8d1d-01f3179877bd-kube-api-access-sv4mn\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866d7f22-cadb-45f7-8d1d-01f3179877bd-tls-certs\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-home\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-tmp-dir\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516499 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-dshm\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516555 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-model-cache\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516611 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-home\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.516666 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.516638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-tmp-dir\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.518684 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.518645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-dshm\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.518834 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.518767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866d7f22-cadb-45f7-8d1d-01f3179877bd-tls-certs\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.525646 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.525624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4mn\" (UniqueName: \"kubernetes.io/projected/866d7f22-cadb-45f7-8d1d-01f3179877bd-kube-api-access-sv4mn\") pod \"auth-test-model-0a9008-kserve-6695cd4747-nfl4r\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.561285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.561263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:45.839034 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:45.839008    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"]
Sep 16 12:29:45.840697 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:29:45.840670    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod866d7f22_cadb_45f7_8d1d_01f3179877bd.slice/crio-0ecb39da163f23c9ce96add647cd703a29d764572b8748a9530fe4408efe7456 WatchSource:0}: Error finding container 0ecb39da163f23c9ce96add647cd703a29d764572b8748a9530fe4408efe7456: Status 404 returned error can't find the container with id 0ecb39da163f23c9ce96add647cd703a29d764572b8748a9530fe4408efe7456
Sep 16 12:29:46.801441 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:46.801405    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r" event={"ID":"866d7f22-cadb-45f7-8d1d-01f3179877bd","Type":"ContainerStarted","Data":"f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585"}
Sep 16 12:29:46.801840 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:46.801451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r" event={"ID":"866d7f22-cadb-45f7-8d1d-01f3179877bd","Type":"ContainerStarted","Data":"0ecb39da163f23c9ce96add647cd703a29d764572b8748a9530fe4408efe7456"}
Sep 16 12:29:46.818052 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:46.817993    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r" podStartSLOduration=1.817978807 podStartE2EDuration="1.817978807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:29:46.816011297 +0000 UTC m=+1582.951333268" watchObservedRunningTime="2026-09-16 12:29:46.817978807 +0000 UTC m=+1582.953300778"
Sep 16 12:29:55.561744 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:55.561710    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:55.561744 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:55.561752    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:55.573805 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:55.573770    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:29:55.861344 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:29:55.861325    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:30:00.144517 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.144479    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"]
Sep 16 12:30:00.148065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.148041    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p"]
Sep 16 12:30:00.148211 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.148150    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"
Sep 16 12:30:00.150770 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.150749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:30:00.151370 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.151353    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"]
Sep 16 12:30:00.151485 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.151466    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p"
Sep 16 12:30:00.154554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.154540    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"]
Sep 16 12:30:00.154657 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.154640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"
Sep 16 12:30:00.158268 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.158248    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"]
Sep 16 12:30:00.158473 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.158458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"
Sep 16 12:30:00.161421 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.161358    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"]
Sep 16 12:30:00.162361 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.162340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"
Sep 16 12:30:00.164072 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.164049    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"]
Sep 16 12:30:00.166838 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.166820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p"]
Sep 16 12:30:00.175279 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.172629    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"]
Sep 16 12:30:00.175810 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.175737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p"
Sep 16 12:30:00.177248 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.177102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"
Sep 16 12:30:00.190165 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.189493    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"]
Sep 16 12:30:00.223448 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.223354    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"
Sep 16 12:30:00.230114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.229985    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"
Sep 16 12:30:00.243986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.240187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"
Sep 16 12:30:00.463726 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.463691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"]
Sep 16 12:30:00.465068 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:30:00.465045    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84184f5_80a9_4ff5_8791_2d9f4cff0ff4.slice/crio-2be8c740876fa596b04b9fa258db95f0ce80401288b561016537166a48b613a1 WatchSource:0}: Error finding container 2be8c740876fa596b04b9fa258db95f0ce80401288b561016537166a48b613a1: Status 404 returned error can't find the container with id 2be8c740876fa596b04b9fa258db95f0ce80401288b561016537166a48b613a1
Sep 16 12:30:00.480011 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.479986    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p"]
Sep 16 12:30:00.481384 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:30:00.481348    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebad9f8_8623_4140_ada5_004a465ef72a.slice/crio-0c5e097c048d89d90177d6f114d4670eecf2378e1bacb674c9cc614b716ee306 WatchSource:0}: Error finding container 0c5e097c048d89d90177d6f114d4670eecf2378e1bacb674c9cc614b716ee306: Status 404 returned error can't find the container with id 0c5e097c048d89d90177d6f114d4670eecf2378e1bacb674c9cc614b716ee306
Sep 16 12:30:00.532623 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.532600    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"]
Sep 16 12:30:00.534334 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:30:00.534307    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60f9cbc_0bd3_48d1_ab1e_a7413ee102d1.slice/crio-98771bc11d85b6b113ae549041166d2c628d768d25a31e10384714036e4e6bba WatchSource:0}: Error finding container 98771bc11d85b6b113ae549041166d2c628d768d25a31e10384714036e4e6bba: Status 404 returned error can't find the container with id 98771bc11d85b6b113ae549041166d2c628d768d25a31e10384714036e4e6bba
Sep 16 12:30:00.563616 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.563590    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"]
Sep 16 12:30:00.565988 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:30:00.565963    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291d3def_3e9e_4e0b_8065_79182a3cb759.slice/crio-4c2086b6fcd69a73e03a4a36a46ec2e37b4d06b46573d72fd39f73873cfca795 WatchSource:0}: Error finding container 4c2086b6fcd69a73e03a4a36a46ec2e37b4d06b46573d72fd39f73873cfca795: Status 404 returned error can't find the container with id 4c2086b6fcd69a73e03a4a36a46ec2e37b4d06b46573d72fd39f73873cfca795
Sep 16 12:30:00.566176 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.566152    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"]
Sep 16 12:30:00.567267 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:30:00.567247    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34407539_9364_4e97_b635_85acd0f42588.slice/crio-7c122a519b95210a1bb0a254c68ad6b2c13737f7c4f36e6e6ccc3b4923cbd2c6 WatchSource:0}: Error finding container 7c122a519b95210a1bb0a254c68ad6b2c13737f7c4f36e6e6ccc3b4923cbd2c6: Status 404 returned error can't find the container with id 7c122a519b95210a1bb0a254c68ad6b2c13737f7c4f36e6e6ccc3b4923cbd2c6
Sep 16 12:30:00.886568 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.886539    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl" event={"ID":"a84184f5-80a9-4ff5-8791-2d9f4cff0ff4","Type":"ContainerStarted","Data":"3c64207c212dea2dccc4eaf0795f0058fa1deef5ad2e92f3cba7215ebd93ec03"}
Sep 16 12:30:00.886683 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.886589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl" event={"ID":"a84184f5-80a9-4ff5-8791-2d9f4cff0ff4","Type":"ContainerStarted","Data":"2be8c740876fa596b04b9fa258db95f0ce80401288b561016537166a48b613a1"}
Sep 16 12:30:00.887846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.887815    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5" event={"ID":"291d3def-3e9e-4e0b-8065-79182a3cb759","Type":"ContainerStarted","Data":"4c2086b6fcd69a73e03a4a36a46ec2e37b4d06b46573d72fd39f73873cfca795"}
Sep 16 12:30:00.889045 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.889018    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx" event={"ID":"d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1","Type":"ContainerStarted","Data":"98771bc11d85b6b113ae549041166d2c628d768d25a31e10384714036e4e6bba"}
Sep 16 12:30:00.890572 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.890548    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p" event={"ID":"9ebad9f8-8623-4140-ada5-004a465ef72a","Type":"ContainerStarted","Data":"edc9e571379554e66162494d524ce2aa5bcefa93195bdffac36a48abcb6bc2c3"}
Sep 16 12:30:00.890682 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.890601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p" event={"ID":"9ebad9f8-8623-4140-ada5-004a465ef72a","Type":"ContainerStarted","Data":"0c5e097c048d89d90177d6f114d4670eecf2378e1bacb674c9cc614b716ee306"}
Sep 16 12:30:00.891829 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.891805    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl" event={"ID":"34407539-9364-4e97-b635-85acd0f42588","Type":"ContainerStarted","Data":"7c122a519b95210a1bb0a254c68ad6b2c13737f7c4f36e6e6ccc3b4923cbd2c6"}
Sep 16 12:30:00.906438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.906399    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl" podStartSLOduration=0.906385298 podStartE2EDuration="906.385298ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:00.904146015 +0000 UTC m=+1597.039467988" watchObservedRunningTime="2026-09-16 12:30:00.906385298 +0000 UTC m=+1597.041707269"
Sep 16 12:30:00.919708 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:00.919616    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p" podStartSLOduration=0.919602446 podStartE2EDuration="919.602446ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:00.917381394 +0000 UTC m=+1597.052703365" watchObservedRunningTime="2026-09-16 12:30:00.919602446 +0000 UTC m=+1597.054924416"
Sep 16 12:30:01.903449 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.903414    2560 generic.go:356] "Generic (PLEG): container finished" podID="d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1" containerID="a9647168c504f313e961852a94acce99804989ab6561e28eb1c0b280f1402ae9" exitCode=0
Sep 16 12:30:01.903855 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.903499    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx" event={"ID":"d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1","Type":"ContainerDied","Data":"a9647168c504f313e961852a94acce99804989ab6561e28eb1c0b280f1402ae9"}
Sep 16 12:30:01.905349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.905323    2560 generic.go:356] "Generic (PLEG): container finished" podID="9ebad9f8-8623-4140-ada5-004a465ef72a" containerID="edc9e571379554e66162494d524ce2aa5bcefa93195bdffac36a48abcb6bc2c3" exitCode=0
Sep 16 12:30:01.905505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.905413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p" event={"ID":"9ebad9f8-8623-4140-ada5-004a465ef72a","Type":"ContainerDied","Data":"edc9e571379554e66162494d524ce2aa5bcefa93195bdffac36a48abcb6bc2c3"}
Sep 16 12:30:01.907194 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.907166    2560 generic.go:356] "Generic (PLEG): container finished" podID="34407539-9364-4e97-b635-85acd0f42588" containerID="97b9076ba7083d20612be9d81bb33ac13574d04a6ad6994747b3542ab224fbb9" exitCode=0
Sep 16 12:30:01.907350 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.907299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl" event={"ID":"34407539-9364-4e97-b635-85acd0f42588","Type":"ContainerDied","Data":"97b9076ba7083d20612be9d81bb33ac13574d04a6ad6994747b3542ab224fbb9"}
Sep 16 12:30:01.909367 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.909343    2560 generic.go:356] "Generic (PLEG): container finished" podID="a84184f5-80a9-4ff5-8791-2d9f4cff0ff4" containerID="3c64207c212dea2dccc4eaf0795f0058fa1deef5ad2e92f3cba7215ebd93ec03" exitCode=0
Sep 16 12:30:01.909520 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.909432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl" event={"ID":"a84184f5-80a9-4ff5-8791-2d9f4cff0ff4","Type":"ContainerDied","Data":"3c64207c212dea2dccc4eaf0795f0058fa1deef5ad2e92f3cba7215ebd93ec03"}
Sep 16 12:30:01.911463 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:01.911442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5" event={"ID":"291d3def-3e9e-4e0b-8065-79182a3cb759","Type":"ContainerStarted","Data":"a081b2031d905cb5e7d11545ae62240da24bd49bf82f0fe4a401291e1fc688cd"}
Sep 16 12:30:02.006414 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:02.006347    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5" podStartSLOduration=2.006323743 podStartE2EDuration="2.006323743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:02.003646469 +0000 UTC m=+1598.138968442" watchObservedRunningTime="2026-09-16 12:30:02.006323743 +0000 UTC m=+1598.141645716"
Sep 16 12:30:02.919193 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:02.919157    2560 generic.go:356] "Generic (PLEG): container finished" podID="291d3def-3e9e-4e0b-8065-79182a3cb759" containerID="a081b2031d905cb5e7d11545ae62240da24bd49bf82f0fe4a401291e1fc688cd" exitCode=0
Sep 16 12:30:02.919588 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:02.919238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5" event={"ID":"291d3def-3e9e-4e0b-8065-79182a3cb759","Type":"ContainerDied","Data":"a081b2031d905cb5e7d11545ae62240da24bd49bf82f0fe4a401291e1fc688cd"}
Sep 16 12:30:03.104834 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.104782    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"
Sep 16 12:30:03.156524 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.156502    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p"
Sep 16 12:30:03.159858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.159837    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"
Sep 16 12:30:03.163397 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.163368    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"
Sep 16 12:30:03.925919 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.925891    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"
Sep 16 12:30:03.926336 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.925917    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl" event={"ID":"a84184f5-80a9-4ff5-8791-2d9f4cff0ff4","Type":"ContainerDied","Data":"2be8c740876fa596b04b9fa258db95f0ce80401288b561016537166a48b613a1"}
Sep 16 12:30:03.926336 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.925969    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be8c740876fa596b04b9fa258db95f0ce80401288b561016537166a48b613a1"
Sep 16 12:30:03.927332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.927314    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"
Sep 16 12:30:03.927459 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.927325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx" event={"ID":"d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1","Type":"ContainerDied","Data":"98771bc11d85b6b113ae549041166d2c628d768d25a31e10384714036e4e6bba"}
Sep 16 12:30:03.927459 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.927389    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98771bc11d85b6b113ae549041166d2c628d768d25a31e10384714036e4e6bba"
Sep 16 12:30:03.928671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.928645    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p"
Sep 16 12:30:03.928826 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.928649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826030-fqj7p" event={"ID":"9ebad9f8-8623-4140-ada5-004a465ef72a","Type":"ContainerDied","Data":"0c5e097c048d89d90177d6f114d4670eecf2378e1bacb674c9cc614b716ee306"}
Sep 16 12:30:03.928826 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.928781    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5e097c048d89d90177d6f114d4670eecf2378e1bacb674c9cc614b716ee306"
Sep 16 12:30:03.930126 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.930109    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"
Sep 16 12:30:03.930215 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.930142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl" event={"ID":"34407539-9364-4e97-b635-85acd0f42588","Type":"ContainerDied","Data":"7c122a519b95210a1bb0a254c68ad6b2c13737f7c4f36e6e6ccc3b4923cbd2c6"}
Sep 16 12:30:03.930215 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:03.930183    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c122a519b95210a1bb0a254c68ad6b2c13737f7c4f36e6e6ccc3b4923cbd2c6"
Sep 16 12:30:04.074605 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:04.074575    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"
Sep 16 12:30:04.936802 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:04.936763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5" event={"ID":"291d3def-3e9e-4e0b-8065-79182a3cb759","Type":"ContainerDied","Data":"4c2086b6fcd69a73e03a4a36a46ec2e37b4d06b46573d72fd39f73873cfca795"}
Sep 16 12:30:04.937151 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:04.936815    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"
Sep 16 12:30:04.937151 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:04.936844    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2086b6fcd69a73e03a4a36a46ec2e37b4d06b46573d72fd39f73873cfca795"
Sep 16 12:30:11.044828 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.044781    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"]
Sep 16 12:30:11.045412 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045390    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84184f5-80a9-4ff5-8791-2d9f4cff0ff4" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045414    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84184f5-80a9-4ff5-8791-2d9f4cff0ff4" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045436    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebad9f8-8623-4140-ada5-004a465ef72a" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045444    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebad9f8-8623-4140-ada5-004a465ef72a" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045453    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045460    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045474    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34407539-9364-4e97-b635-85acd0f42588" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045482    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="34407539-9364-4e97-b635-85acd0f42588" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045502    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291d3def-3e9e-4e0b-8065-79182a3cb759" containerName="cleanup"
Sep 16 12:30:11.045513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045509    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="291d3def-3e9e-4e0b-8065-79182a3cb759" containerName="cleanup"
Sep 16 12:30:11.045969 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045632    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1" containerName="cleanup"
Sep 16 12:30:11.045969 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045643    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="291d3def-3e9e-4e0b-8065-79182a3cb759" containerName="cleanup"
Sep 16 12:30:11.045969 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045653    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84184f5-80a9-4ff5-8791-2d9f4cff0ff4" containerName="cleanup"
Sep 16 12:30:11.045969 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045665    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="34407539-9364-4e97-b635-85acd0f42588" containerName="cleanup"
Sep 16 12:30:11.045969 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.045675    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebad9f8-8623-4140-ada5-004a465ef72a" containerName="cleanup"
Sep 16 12:30:11.049718 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.049694    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.052886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.052861    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-68f7cd\"/\"distinct-2-w4-68f7cd-kserve-self-signed-certs\""
Sep 16 12:30:11.058528 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.058505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"]
Sep 16 12:30:11.147605 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.147564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-home\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.147772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.147676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-model-cache\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.147772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.147735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-dshm\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.147893 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.147857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-tmp-dir\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.147943 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.147893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvjb\" (UniqueName: \"kubernetes.io/projected/73efacf5-21cc-4b18-8c6e-446d77915866-kube-api-access-4vvjb\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.147943 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.147921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73efacf5-21cc-4b18-8c6e-446d77915866-tls-certs\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.248949 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.248918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-home\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.248967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-model-cache\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.249017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-dshm\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.249096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-tmp-dir\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.249127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvjb\" (UniqueName: \"kubernetes.io/projected/73efacf5-21cc-4b18-8c6e-446d77915866-kube-api-access-4vvjb\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.249158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73efacf5-21cc-4b18-8c6e-446d77915866-tls-certs\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249372 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.249347    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-home\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249424 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.249386    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-model-cache\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.249503 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.249483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-tmp-dir\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.251541 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.251520    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-dshm\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.251699 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.251683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73efacf5-21cc-4b18-8c6e-446d77915866-tls-certs\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.257547 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.257523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvjb\" (UniqueName: \"kubernetes.io/projected/73efacf5-21cc-4b18-8c6e-446d77915866-kube-api-access-4vvjb\") pod \"distinct-2-w4-68f7cd-kserve-78955d7958-trnxk\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") " pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.361082 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.361048    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:11.634449 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.634419    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"]
Sep 16 12:30:11.636235 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:30:11.636207    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73efacf5_21cc_4b18_8c6e_446d77915866.slice/crio-477266280aa7c24a580acaeb8e847e1f3e43ec5fa06e84ef9e20e1f30e66def3 WatchSource:0}: Error finding container 477266280aa7c24a580acaeb8e847e1f3e43ec5fa06e84ef9e20e1f30e66def3: Status 404 returned error can't find the container with id 477266280aa7c24a580acaeb8e847e1f3e43ec5fa06e84ef9e20e1f30e66def3
Sep 16 12:30:11.978354 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.978269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk" event={"ID":"73efacf5-21cc-4b18-8c6e-446d77915866","Type":"ContainerStarted","Data":"97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb"}
Sep 16 12:30:11.978354 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.978310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk" event={"ID":"73efacf5-21cc-4b18-8c6e-446d77915866","Type":"ContainerStarted","Data":"477266280aa7c24a580acaeb8e847e1f3e43ec5fa06e84ef9e20e1f30e66def3"}
Sep 16 12:30:11.997630 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:11.997581    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk" podStartSLOduration=0.997552113 podStartE2EDuration="997.552113ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:11.996462868 +0000 UTC m=+1608.131784839" watchObservedRunningTime="2026-09-16 12:30:11.997552113 +0000 UTC m=+1608.132874118"
Sep 16 12:30:21.362171 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:21.362074    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:21.362171 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:21.362166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:21.374985 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:21.374961    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:30:22.058675 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:30:22.058645    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:31:15.625653 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.625610    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"]
Sep 16 12:31:15.634972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.634947    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:15.641396 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.641375    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8641cd\""
Sep 16 12:31:15.641598 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.641577    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8641cd-dockercfg-g56mz\""
Sep 16 12:31:15.678689 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.678667    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"]
Sep 16 12:31:15.744710 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.744679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118c0f39-5592-4165-92eb-4c594cb01941-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:15.744866 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.744757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgxk\" (UniqueName: \"kubernetes.io/projected/118c0f39-5592-4165-92eb-4c594cb01941-kube-api-access-6cgxk\") pod \"payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:15.845690 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.845656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgxk\" (UniqueName: \"kubernetes.io/projected/118c0f39-5592-4165-92eb-4c594cb01941-kube-api-access-6cgxk\") pod \"payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:15.845843 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.845733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118c0f39-5592-4165-92eb-4c594cb01941-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:15.846356 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.846337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118c0f39-5592-4165-92eb-4c594cb01941-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:15.860962 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.860932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgxk\" (UniqueName: \"kubernetes.io/projected/118c0f39-5592-4165-92eb-4c594cb01941-kube-api-access-6cgxk\") pod \"payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:15.945874 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:15.945811    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:16.239236 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:16.239203    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"]
Sep 16 12:31:16.239468 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:31:16.239440    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118c0f39_5592_4165_92eb_4c594cb01941.slice/crio-d3eba1fbafb4bd6ee57e9de7642aadfd0d252f54a75efdc49cd2feca5fa61a01 WatchSource:0}: Error finding container d3eba1fbafb4bd6ee57e9de7642aadfd0d252f54a75efdc49cd2feca5fa61a01: Status 404 returned error can't find the container with id d3eba1fbafb4bd6ee57e9de7642aadfd0d252f54a75efdc49cd2feca5fa61a01
Sep 16 12:31:16.378127 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:16.378093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m" event={"ID":"118c0f39-5592-4165-92eb-4c594cb01941","Type":"ContainerStarted","Data":"539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc"}
Sep 16 12:31:16.378275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:16.378150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m" event={"ID":"118c0f39-5592-4165-92eb-4c594cb01941","Type":"ContainerStarted","Data":"d3eba1fbafb4bd6ee57e9de7642aadfd0d252f54a75efdc49cd2feca5fa61a01"}
Sep 16 12:31:16.378340 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:16.378315    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:16.416423 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:16.416368    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m" podStartSLOduration=1.4163498190000001 podStartE2EDuration="1.416349819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:16.413231105 +0000 UTC m=+1672.548553078" watchObservedRunningTime="2026-09-16 12:31:16.416349819 +0000 UTC m=+1672.551671791"
Sep 16 12:31:24.561601 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:24.561568    2560 scope.go:117] "RemoveContainer" containerID="32f35805331afac4230b30fd87acc5487d77ff1177103fc0d62e8844664350bb"
Sep 16 12:31:24.571483 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:24.571463    2560 scope.go:117] "RemoveContainer" containerID="b06ee201382c1b4268aa961969463cd3e0a8441b1671c5db0669de4e43b4c929"
Sep 16 12:31:27.386996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:27.386963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:31:38.726533 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.726499    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"]
Sep 16 12:31:38.730003 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.729986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"
Sep 16 12:31:38.732820 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.732796    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:31:38.733116 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.733091    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:31:38.750057 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.750032    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"]
Sep 16 12:31:38.864802 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.864750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e42b-ff8f-4817-af06-260a1cbf941c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s\" (UID: \"fd81e42b-ff8f-4817-af06-260a1cbf941c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"
Sep 16 12:31:38.965765 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.965735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e42b-ff8f-4817-af06-260a1cbf941c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s\" (UID: \"fd81e42b-ff8f-4817-af06-260a1cbf941c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"
Sep 16 12:31:38.966290 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:38.966273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e42b-ff8f-4817-af06-260a1cbf941c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s\" (UID: \"fd81e42b-ff8f-4817-af06-260a1cbf941c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"
Sep 16 12:31:39.040438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:39.040385    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"
Sep 16 12:31:39.322727 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:39.322686    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"]
Sep 16 12:31:39.326331 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:31:39.326303    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd81e42b_ff8f_4817_af06_260a1cbf941c.slice/crio-44efc7d9efdb226c883b72075f7ae23fdc8e99804f3759439135999ed99a9fe2 WatchSource:0}: Error finding container 44efc7d9efdb226c883b72075f7ae23fdc8e99804f3759439135999ed99a9fe2: Status 404 returned error can't find the container with id 44efc7d9efdb226c883b72075f7ae23fdc8e99804f3759439135999ed99a9fe2
Sep 16 12:31:39.514474 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:39.514444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s" event={"ID":"fd81e42b-ff8f-4817-af06-260a1cbf941c","Type":"ContainerStarted","Data":"44efc7d9efdb226c883b72075f7ae23fdc8e99804f3759439135999ed99a9fe2"}
Sep 16 12:31:40.522520 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:40.522482    2560 generic.go:356] "Generic (PLEG): container finished" podID="fd81e42b-ff8f-4817-af06-260a1cbf941c" containerID="54bef9dc565d94818e27cb20663574829962fbee6c6f1a814e9847a8679cd61e" exitCode=0
Sep 16 12:31:40.523014 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:40.522583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s" event={"ID":"fd81e42b-ff8f-4817-af06-260a1cbf941c","Type":"ContainerDied","Data":"54bef9dc565d94818e27cb20663574829962fbee6c6f1a814e9847a8679cd61e"}
Sep 16 12:31:41.670570 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:41.670546    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"
Sep 16 12:31:41.790247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:41.790175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e42b-ff8f-4817-af06-260a1cbf941c-maas-api-service-ca\") pod \"fd81e42b-ff8f-4817-af06-260a1cbf941c\" (UID: \"fd81e42b-ff8f-4817-af06-260a1cbf941c\") "
Sep 16 12:31:41.790505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:41.790485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd81e42b-ff8f-4817-af06-260a1cbf941c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd81e42b-ff8f-4817-af06-260a1cbf941c" (UID: "fd81e42b-ff8f-4817-af06-260a1cbf941c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:31:41.890927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:41.890902    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd81e42b-ff8f-4817-af06-260a1cbf941c-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:31:42.535165 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:42.535136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s" event={"ID":"fd81e42b-ff8f-4817-af06-260a1cbf941c","Type":"ContainerDied","Data":"44efc7d9efdb226c883b72075f7ae23fdc8e99804f3759439135999ed99a9fe2"}
Sep 16 12:31:42.535320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:42.535172    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"
Sep 16 12:31:42.535320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:42.535179    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44efc7d9efdb226c883b72075f7ae23fdc8e99804f3759439135999ed99a9fe2"
Sep 16 12:31:43.820066 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:43.820027    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"]
Sep 16 12:31:55.420859 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.420770    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"]
Sep 16 12:31:55.421270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.421241    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd81e42b-ff8f-4817-af06-260a1cbf941c" containerName="revoke-keys"
Sep 16 12:31:55.421270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.421253    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd81e42b-ff8f-4817-af06-260a1cbf941c" containerName="revoke-keys"
Sep 16 12:31:55.421346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.421336    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd81e42b-ff8f-4817-af06-260a1cbf941c" containerName="revoke-keys"
Sep 16 12:31:55.423456 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.423439    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:55.437361 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.437342    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-352217\""
Sep 16 12:31:55.440894 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.440879    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-352217-dockercfg-6nbhq\""
Sep 16 12:31:55.471088 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.471065    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"]
Sep 16 12:31:55.514407 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.514380    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"]
Sep 16 12:31:55.517052 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.517034    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:55.552331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.552307    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"]
Sep 16 12:31:55.608528 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.608498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:55.608648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.608576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4f6\" (UniqueName: \"kubernetes.io/projected/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-kube-api-access-vq4f6\") pod \"payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:55.608648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.608608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kv9\" (UniqueName: \"kubernetes.io/projected/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-kube-api-access-97kv9\") pod \"payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:55.608734 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.608716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:55.709836 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.709758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:55.709944 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.709834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:55.710014 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.709999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4f6\" (UniqueName: \"kubernetes.io/projected/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-kube-api-access-vq4f6\") pod \"payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:55.710075 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.710042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kv9\" (UniqueName: \"kubernetes.io/projected/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-kube-api-access-97kv9\") pod \"payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:55.710697 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.710677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:55.710774 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.710679    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:55.729007 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.728983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4f6\" (UniqueName: \"kubernetes.io/projected/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-kube-api-access-vq4f6\") pod \"payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:55.738591 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.738572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kv9\" (UniqueName: \"kubernetes.io/projected/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-kube-api-access-97kv9\") pod \"payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:55.827282 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:55.827250    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:56.034394 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.034321    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:56.115668 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.115631    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"]
Sep 16 12:31:56.122694 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:31:56.122649    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec577a61_8a5a_42a6_a290_b3b84f6b93d5.slice/crio-ac6c76e673e54ac1cc97e9ac33d763cacf963b1d1ceaf45b30d4d53774e9906b WatchSource:0}: Error finding container ac6c76e673e54ac1cc97e9ac33d763cacf963b1d1ceaf45b30d4d53774e9906b: Status 404 returned error can't find the container with id ac6c76e673e54ac1cc97e9ac33d763cacf963b1d1ceaf45b30d4d53774e9906b
Sep 16 12:31:56.333247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.333220    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"]
Sep 16 12:31:56.333982 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:31:56.333951    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777d3db6_9f85_4a46_a3ce_52b9b951dd2d.slice/crio-2ba3634784be51ddf36447757104b3f689699b7b34e0910cf0bce80918c709a3 WatchSource:0}: Error finding container 2ba3634784be51ddf36447757104b3f689699b7b34e0910cf0bce80918c709a3: Status 404 returned error can't find the container with id 2ba3634784be51ddf36447757104b3f689699b7b34e0910cf0bce80918c709a3
Sep 16 12:31:56.621048 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.621012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z" event={"ID":"777d3db6-9f85-4a46-a3ce-52b9b951dd2d","Type":"ContainerStarted","Data":"547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0"}
Sep 16 12:31:56.621481 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.621068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z" event={"ID":"777d3db6-9f85-4a46-a3ce-52b9b951dd2d","Type":"ContainerStarted","Data":"2ba3634784be51ddf36447757104b3f689699b7b34e0910cf0bce80918c709a3"}
Sep 16 12:31:56.621481 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.621096    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:31:56.623043 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.623013    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq" event={"ID":"ec577a61-8a5a-42a6-a290-b3b84f6b93d5","Type":"ContainerStarted","Data":"b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5"}
Sep 16 12:31:56.623194 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.623057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq" event={"ID":"ec577a61-8a5a-42a6-a290-b3b84f6b93d5","Type":"ContainerStarted","Data":"ac6c76e673e54ac1cc97e9ac33d763cacf963b1d1ceaf45b30d4d53774e9906b"}
Sep 16 12:31:56.623194 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.623126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:31:56.669446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:56.669402    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z" podStartSLOduration=1.6693892940000001 podStartE2EDuration="1.669389294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:56.66623198 +0000 UTC m=+1712.801553952" watchObservedRunningTime="2026-09-16 12:31:56.669389294 +0000 UTC m=+1712.804711264"
Sep 16 12:31:58.839917 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:58.839878    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh" podUID="484df775-cdde-4907-b3b1-1a369d374007" containerName="main" containerID="cri-o://94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0" gracePeriod=45
Sep 16 12:31:59.097062 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.097035    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:31:59.142809 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.142751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-model-cache\") pod \"484df775-cdde-4907-b3b1-1a369d374007\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") "
Sep 16 12:31:59.142939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.142816    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484df775-cdde-4907-b3b1-1a369d374007-tls-certs\") pod \"484df775-cdde-4907-b3b1-1a369d374007\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") "
Sep 16 12:31:59.142939 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.142845    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4r7d\" (UniqueName: \"kubernetes.io/projected/484df775-cdde-4907-b3b1-1a369d374007-kube-api-access-v4r7d\") pod \"484df775-cdde-4907-b3b1-1a369d374007\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") "
Sep 16 12:31:59.143072 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.142995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-home\") pod \"484df775-cdde-4907-b3b1-1a369d374007\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") "
Sep 16 12:31:59.143072 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143021    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-dshm\") pod \"484df775-cdde-4907-b3b1-1a369d374007\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") "
Sep 16 12:31:59.143072 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143031    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-model-cache" (OuterVolumeSpecName: "model-cache") pod "484df775-cdde-4907-b3b1-1a369d374007" (UID: "484df775-cdde-4907-b3b1-1a369d374007"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.143224 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143071    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-tmp-dir\") pod \"484df775-cdde-4907-b3b1-1a369d374007\" (UID: \"484df775-cdde-4907-b3b1-1a369d374007\") "
Sep 16 12:31:59.143292 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143260    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-home" (OuterVolumeSpecName: "home") pod "484df775-cdde-4907-b3b1-1a369d374007" (UID: "484df775-cdde-4907-b3b1-1a369d374007"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.143390 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143368    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "484df775-cdde-4907-b3b1-1a369d374007" (UID: "484df775-cdde-4907-b3b1-1a369d374007"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.143754 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143685    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.143754 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143710    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.143754 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.143723    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.145240 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.145215    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484df775-cdde-4907-b3b1-1a369d374007-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "484df775-cdde-4907-b3b1-1a369d374007" (UID: "484df775-cdde-4907-b3b1-1a369d374007"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:59.145346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.145269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-dshm" (OuterVolumeSpecName: "dshm") pod "484df775-cdde-4907-b3b1-1a369d374007" (UID: "484df775-cdde-4907-b3b1-1a369d374007"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.145346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.145333    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484df775-cdde-4907-b3b1-1a369d374007-kube-api-access-v4r7d" (OuterVolumeSpecName: "kube-api-access-v4r7d") pod "484df775-cdde-4907-b3b1-1a369d374007" (UID: "484df775-cdde-4907-b3b1-1a369d374007"). InnerVolumeSpecName "kube-api-access-v4r7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:59.167451 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.167389    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq" podStartSLOduration=4.167370325 podStartE2EDuration="4.167370325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:56.715687131 +0000 UTC m=+1712.851009103" watchObservedRunningTime="2026-09-16 12:31:59.167370325 +0000 UTC m=+1715.302692298"
Sep 16 12:31:59.244861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.244826    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484df775-cdde-4907-b3b1-1a369d374007-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.245008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.244892    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484df775-cdde-4907-b3b1-1a369d374007-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.245008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.244909    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4r7d\" (UniqueName: \"kubernetes.io/projected/484df775-cdde-4907-b3b1-1a369d374007-kube-api-access-v4r7d\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.642674 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.642644    2560 generic.go:356] "Generic (PLEG): container finished" podID="484df775-cdde-4907-b3b1-1a369d374007" containerID="94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0" exitCode=0
Sep 16 12:31:59.642857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.642704    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"
Sep 16 12:31:59.642857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.642722    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh" event={"ID":"484df775-cdde-4907-b3b1-1a369d374007","Type":"ContainerDied","Data":"94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0"}
Sep 16 12:31:59.642857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.642763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh" event={"ID":"484df775-cdde-4907-b3b1-1a369d374007","Type":"ContainerDied","Data":"6e786509dc338860bfd99a54aaed8132f1f42148009729773125059658eae5a2"}
Sep 16 12:31:59.642857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.642779    2560 scope.go:117] "RemoveContainer" containerID="94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0"
Sep 16 12:31:59.652714 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.652697    2560 scope.go:117] "RemoveContainer" containerID="94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0"
Sep 16 12:31:59.653004 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:31:59.652986    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0\": container with ID starting with 94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0 not found: ID does not exist" containerID="94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0"
Sep 16 12:31:59.653062 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.653012    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0"} err="failed to get container status \"94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0\": rpc error: code = NotFound desc = could not find container \"94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0\": container with ID starting with 94eb69851b7c0fccf24cf07b71e4b9e216064566b178c9a755b89d0315a2d3e0 not found: ID does not exist"
Sep 16 12:31:59.678802 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.678769    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"]
Sep 16 12:31:59.702413 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:31:59.702393    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/simulator-w0-739ee7-kserve-8fc5bbb9-47hdh"]
Sep 16 12:32:00.325490 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:00.325455    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484df775-cdde-4907-b3b1-1a369d374007" path="/var/lib/kubelet/pods/484df775-cdde-4907-b3b1-1a369d374007/volumes"
Sep 16 12:32:01.877580 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:01.877549    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"]
Sep 16 12:32:01.878086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:01.877750    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248" podUID="22ed33c5-b4fa-4bc5-9357-4a12399784e1" containerName="payload-processing" containerID="cri-o://d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992" gracePeriod=30
Sep 16 12:32:01.900009 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:01.899983    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"]
Sep 16 12:32:01.900248 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:01.900214    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v" podUID="f98864b0-bcec-46b8-95f1-fba92b3234ed" containerName="payload-pre-processing" containerID="cri-o://d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769" gracePeriod=30
Sep 16 12:32:02.175403 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.175381    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:32:02.179358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.179341    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:32:02.271510 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.271469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t9qn\" (UniqueName: \"kubernetes.io/projected/f98864b0-bcec-46b8-95f1-fba92b3234ed-kube-api-access-6t9qn\") pod \"f98864b0-bcec-46b8-95f1-fba92b3234ed\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") "
Sep 16 12:32:02.271702 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.271538    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wgbp\" (UniqueName: \"kubernetes.io/projected/22ed33c5-b4fa-4bc5-9357-4a12399784e1-kube-api-access-7wgbp\") pod \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") "
Sep 16 12:32:02.271702 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.271649    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ed33c5-b4fa-4bc5-9357-4a12399784e1-plugins-config-volume\") pod \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\" (UID: \"22ed33c5-b4fa-4bc5-9357-4a12399784e1\") "
Sep 16 12:32:02.271860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.271732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98864b0-bcec-46b8-95f1-fba92b3234ed-plugins-config-volume\") pod \"f98864b0-bcec-46b8-95f1-fba92b3234ed\" (UID: \"f98864b0-bcec-46b8-95f1-fba92b3234ed\") "
Sep 16 12:32:02.272373 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.272079    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ed33c5-b4fa-4bc5-9357-4a12399784e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22ed33c5-b4fa-4bc5-9357-4a12399784e1" (UID: "22ed33c5-b4fa-4bc5-9357-4a12399784e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:02.272373 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.272207    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22ed33c5-b4fa-4bc5-9357-4a12399784e1-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.272744 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.272593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98864b0-bcec-46b8-95f1-fba92b3234ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f98864b0-bcec-46b8-95f1-fba92b3234ed" (UID: "f98864b0-bcec-46b8-95f1-fba92b3234ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:02.274222 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.274195    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed33c5-b4fa-4bc5-9357-4a12399784e1-kube-api-access-7wgbp" (OuterVolumeSpecName: "kube-api-access-7wgbp") pod "22ed33c5-b4fa-4bc5-9357-4a12399784e1" (UID: "22ed33c5-b4fa-4bc5-9357-4a12399784e1"). InnerVolumeSpecName "kube-api-access-7wgbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:02.274524 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.274492    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98864b0-bcec-46b8-95f1-fba92b3234ed-kube-api-access-6t9qn" (OuterVolumeSpecName: "kube-api-access-6t9qn") pod "f98864b0-bcec-46b8-95f1-fba92b3234ed" (UID: "f98864b0-bcec-46b8-95f1-fba92b3234ed"). InnerVolumeSpecName "kube-api-access-6t9qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:02.356520 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.356467    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"]
Sep 16 12:32:02.361853 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.361818    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-739ee7-nv99s"]
Sep 16 12:32:02.373423 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.373405    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98864b0-bcec-46b8-95f1-fba92b3234ed-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.373505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.373425    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9qn\" (UniqueName: \"kubernetes.io/projected/f98864b0-bcec-46b8-95f1-fba92b3234ed-kube-api-access-6t9qn\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.373505 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.373439    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wgbp\" (UniqueName: \"kubernetes.io/projected/22ed33c5-b4fa-4bc5-9357-4a12399784e1-kube-api-access-7wgbp\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:02.665653 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.665622    2560 generic.go:356] "Generic (PLEG): container finished" podID="22ed33c5-b4fa-4bc5-9357-4a12399784e1" containerID="d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992" exitCode=0
Sep 16 12:32:02.665846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.665670    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"
Sep 16 12:32:02.665846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.665698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248" event={"ID":"22ed33c5-b4fa-4bc5-9357-4a12399784e1","Type":"ContainerDied","Data":"d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992"}
Sep 16 12:32:02.665846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.665757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248" event={"ID":"22ed33c5-b4fa-4bc5-9357-4a12399784e1","Type":"ContainerDied","Data":"50f348f28a31e098dbef87f4c6ba86264ed834b64d1f7dacd134b7ba37793c6d"}
Sep 16 12:32:02.665846 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.665815    2560 scope.go:117] "RemoveContainer" containerID="d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992"
Sep 16 12:32:02.667408 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.667336    2560 generic.go:356] "Generic (PLEG): container finished" podID="f98864b0-bcec-46b8-95f1-fba92b3234ed" containerID="d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769" exitCode=0
Sep 16 12:32:02.667408 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.667396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v" event={"ID":"f98864b0-bcec-46b8-95f1-fba92b3234ed","Type":"ContainerDied","Data":"d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769"}
Sep 16 12:32:02.667581 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.667439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v" event={"ID":"f98864b0-bcec-46b8-95f1-fba92b3234ed","Type":"ContainerDied","Data":"0d77b6d142d621a1282022cc6083478653a1c1adae29a4ea57e772bdda6daffa"}
Sep 16 12:32:02.667581 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.667380    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"
Sep 16 12:32:02.675706 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.675688    2560 scope.go:117] "RemoveContainer" containerID="d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992"
Sep 16 12:32:02.676053 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:32:02.676035    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992\": container with ID starting with d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992 not found: ID does not exist" containerID="d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992"
Sep 16 12:32:02.676134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.676061    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992"} err="failed to get container status \"d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992\": rpc error: code = NotFound desc = could not find container \"d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992\": container with ID starting with d5a94c073f6ad957599758978fb606292bd9816700f97fd9c2b1cb78d3cfe992 not found: ID does not exist"
Sep 16 12:32:02.676134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.676077    2560 scope.go:117] "RemoveContainer" containerID="d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769"
Sep 16 12:32:02.684269 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.684252    2560 scope.go:117] "RemoveContainer" containerID="d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769"
Sep 16 12:32:02.684551 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:32:02.684531    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769\": container with ID starting with d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769 not found: ID does not exist" containerID="d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769"
Sep 16 12:32:02.684611 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.684562    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769"} err="failed to get container status \"d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769\": rpc error: code = NotFound desc = could not find container \"d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769\": container with ID starting with d2c4d7a77dd860e57eda6c540c375a1999d2c06707263d42a4890e1831042769 not found: ID does not exist"
Sep 16 12:32:02.720471 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.720439    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"]
Sep 16 12:32:02.722621 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.722602    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-739ee7-bfb8dbf4b-77h7v"]
Sep 16 12:32:02.748567 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.748541    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"]
Sep 16 12:32:02.759734 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:02.759705    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-739ee7-567bbbbfbc-bv248"]
Sep 16 12:32:04.325268 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:04.325227    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22ed33c5-b4fa-4bc5-9357-4a12399784e1" path="/var/lib/kubelet/pods/22ed33c5-b4fa-4bc5-9357-4a12399784e1/volumes"
Sep 16 12:32:04.325759 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:04.325738    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f98864b0-bcec-46b8-95f1-fba92b3234ed" path="/var/lib/kubelet/pods/f98864b0-bcec-46b8-95f1-fba92b3234ed/volumes"
Sep 16 12:32:04.326118 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:04.326100    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd81e42b-ff8f-4817-af06-260a1cbf941c" path="/var/lib/kubelet/pods/fd81e42b-ff8f-4817-af06-260a1cbf941c/volumes"
Sep 16 12:32:07.631634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:07.631608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:32:07.632073 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:07.631671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:32:13.265204 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265170    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"]
Sep 16 12:32:13.265629 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265617    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98864b0-bcec-46b8-95f1-fba92b3234ed" containerName="payload-pre-processing"
Sep 16 12:32:13.265675 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265631    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98864b0-bcec-46b8-95f1-fba92b3234ed" containerName="payload-pre-processing"
Sep 16 12:32:13.265675 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265641    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ed33c5-b4fa-4bc5-9357-4a12399784e1" containerName="payload-processing"
Sep 16 12:32:13.265675 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265647    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ed33c5-b4fa-4bc5-9357-4a12399784e1" containerName="payload-processing"
Sep 16 12:32:13.265675 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265657    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484df775-cdde-4907-b3b1-1a369d374007" containerName="main"
Sep 16 12:32:13.265675 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265665    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="484df775-cdde-4907-b3b1-1a369d374007" containerName="main"
Sep 16 12:32:13.265857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265746    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98864b0-bcec-46b8-95f1-fba92b3234ed" containerName="payload-pre-processing"
Sep 16 12:32:13.265857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265755    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ed33c5-b4fa-4bc5-9357-4a12399784e1" containerName="payload-processing"
Sep 16 12:32:13.265857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.265762    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="484df775-cdde-4907-b3b1-1a369d374007" containerName="main"
Sep 16 12:32:13.268808 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.268769    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"
Sep 16 12:32:13.271956 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.271929    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:32:13.272104 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.272030    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:32:13.292795 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.292769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"]
Sep 16 12:32:13.384973 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.384939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7beb628-8d69-440b-8cee-b33026be56a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8641cd-tz582\" (UID: \"d7beb628-8d69-440b-8cee-b33026be56a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"
Sep 16 12:32:13.486499 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.486464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7beb628-8d69-440b-8cee-b33026be56a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8641cd-tz582\" (UID: \"d7beb628-8d69-440b-8cee-b33026be56a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"
Sep 16 12:32:13.487062 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.487044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7beb628-8d69-440b-8cee-b33026be56a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8641cd-tz582\" (UID: \"d7beb628-8d69-440b-8cee-b33026be56a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"
Sep 16 12:32:13.579570 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.579497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"
Sep 16 12:32:13.881028 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:32:13.881001    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7beb628_8d69_440b_8cee_b33026be56a0.slice/crio-3d6788607f633a93496d136e8797cc91d2ad85522a989ad2d50802e6d5bfee8d WatchSource:0}: Error finding container 3d6788607f633a93496d136e8797cc91d2ad85522a989ad2d50802e6d5bfee8d: Status 404 returned error can't find the container with id 3d6788607f633a93496d136e8797cc91d2ad85522a989ad2d50802e6d5bfee8d
Sep 16 12:32:13.885060 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:13.885038    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"]
Sep 16 12:32:14.747143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:14.747094    2560 generic.go:356] "Generic (PLEG): container finished" podID="d7beb628-8d69-440b-8cee-b33026be56a0" containerID="904701144c10549f840853ccedbc8b20ceec1217a8d1b73a2a02c601613349fb" exitCode=0
Sep 16 12:32:14.747538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:14.747172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582" event={"ID":"d7beb628-8d69-440b-8cee-b33026be56a0","Type":"ContainerDied","Data":"904701144c10549f840853ccedbc8b20ceec1217a8d1b73a2a02c601613349fb"}
Sep 16 12:32:14.747538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:14.747273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582" event={"ID":"d7beb628-8d69-440b-8cee-b33026be56a0","Type":"ContainerStarted","Data":"3d6788607f633a93496d136e8797cc91d2ad85522a989ad2d50802e6d5bfee8d"}
Sep 16 12:32:15.891245 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:15.891222    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"
Sep 16 12:32:16.016812 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:16.016721    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7beb628-8d69-440b-8cee-b33026be56a0-maas-api-service-ca\") pod \"d7beb628-8d69-440b-8cee-b33026be56a0\" (UID: \"d7beb628-8d69-440b-8cee-b33026be56a0\") "
Sep 16 12:32:16.017052 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:16.017028    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7beb628-8d69-440b-8cee-b33026be56a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7beb628-8d69-440b-8cee-b33026be56a0" (UID: "d7beb628-8d69-440b-8cee-b33026be56a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:16.017148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:16.017136    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7beb628-8d69-440b-8cee-b33026be56a0-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:16.761244 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:16.761214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"
Sep 16 12:32:16.761418 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:16.761252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582" event={"ID":"d7beb628-8d69-440b-8cee-b33026be56a0","Type":"ContainerDied","Data":"3d6788607f633a93496d136e8797cc91d2ad85522a989ad2d50802e6d5bfee8d"}
Sep 16 12:32:16.761418 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:16.761299    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d6788607f633a93496d136e8797cc91d2ad85522a989ad2d50802e6d5bfee8d"
Sep 16 12:32:31.581888 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.581844    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"]
Sep 16 12:32:31.582477 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.582088    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m" podUID="118c0f39-5592-4165-92eb-4c594cb01941" containerName="payload-pre-processing" containerID="cri-o://539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc" gracePeriod=30
Sep 16 12:32:31.782689 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.782652    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"]
Sep 16 12:32:31.788715 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.788689    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8641cd-tz582"]
Sep 16 12:32:31.826963 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.826941    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:32:31.864890 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.864860    2560 generic.go:356] "Generic (PLEG): container finished" podID="118c0f39-5592-4165-92eb-4c594cb01941" containerID="539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc" exitCode=0
Sep 16 12:32:31.865077 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.864911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m" event={"ID":"118c0f39-5592-4165-92eb-4c594cb01941","Type":"ContainerDied","Data":"539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc"}
Sep 16 12:32:31.865077 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.864920    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"
Sep 16 12:32:31.865077 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.864980    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m" event={"ID":"118c0f39-5592-4165-92eb-4c594cb01941","Type":"ContainerDied","Data":"d3eba1fbafb4bd6ee57e9de7642aadfd0d252f54a75efdc49cd2feca5fa61a01"}
Sep 16 12:32:31.865077 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.865004    2560 scope.go:117] "RemoveContainer" containerID="539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc"
Sep 16 12:32:31.875352 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.875327    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118c0f39-5592-4165-92eb-4c594cb01941-plugins-config-volume\") pod \"118c0f39-5592-4165-92eb-4c594cb01941\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") "
Sep 16 12:32:31.875452 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.875373    2560 scope.go:117] "RemoveContainer" containerID="539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc"
Sep 16 12:32:31.875452 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.875425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cgxk\" (UniqueName: \"kubernetes.io/projected/118c0f39-5592-4165-92eb-4c594cb01941-kube-api-access-6cgxk\") pod \"118c0f39-5592-4165-92eb-4c594cb01941\" (UID: \"118c0f39-5592-4165-92eb-4c594cb01941\") "
Sep 16 12:32:31.875740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.875713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118c0f39-5592-4165-92eb-4c594cb01941-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "118c0f39-5592-4165-92eb-4c594cb01941" (UID: "118c0f39-5592-4165-92eb-4c594cb01941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:31.876058 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:32:31.876033    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc\": container with ID starting with 539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc not found: ID does not exist" containerID="539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc"
Sep 16 12:32:31.876121 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.876069    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc"} err="failed to get container status \"539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc\": rpc error: code = NotFound desc = could not find container \"539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc\": container with ID starting with 539f9c86a41779b21f0553229de41f220966aa1343ed24101141475f6f7879dc not found: ID does not exist"
Sep 16 12:32:31.877651 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.877629    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118c0f39-5592-4165-92eb-4c594cb01941-kube-api-access-6cgxk" (OuterVolumeSpecName: "kube-api-access-6cgxk") pod "118c0f39-5592-4165-92eb-4c594cb01941" (UID: "118c0f39-5592-4165-92eb-4c594cb01941"). InnerVolumeSpecName "kube-api-access-6cgxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:31.976316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.976277    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/118c0f39-5592-4165-92eb-4c594cb01941-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.976316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:31.976312    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgxk\" (UniqueName: \"kubernetes.io/projected/118c0f39-5592-4165-92eb-4c594cb01941-kube-api-access-6cgxk\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.191591 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:32.191555    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"]
Sep 16 12:32:32.197437 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:32.197412    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8641cd-55dcf449fd-ggv9m"]
Sep 16 12:32:32.325671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:32.325642    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118c0f39-5592-4165-92eb-4c594cb01941" path="/var/lib/kubelet/pods/118c0f39-5592-4165-92eb-4c594cb01941/volumes"
Sep 16 12:32:32.326013 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:32.325998    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7beb628-8d69-440b-8cee-b33026be56a0" path="/var/lib/kubelet/pods/d7beb628-8d69-440b-8cee-b33026be56a0/volumes"
Sep 16 12:32:40.546558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.546528    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"]
Sep 16 12:32:40.546972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.546958    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7beb628-8d69-440b-8cee-b33026be56a0" containerName="revoke-keys"
Sep 16 12:32:40.547018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.546974    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7beb628-8d69-440b-8cee-b33026be56a0" containerName="revoke-keys"
Sep 16 12:32:40.547018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.546997    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118c0f39-5592-4165-92eb-4c594cb01941" containerName="payload-pre-processing"
Sep 16 12:32:40.547018 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.547003    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="118c0f39-5592-4165-92eb-4c594cb01941" containerName="payload-pre-processing"
Sep 16 12:32:40.547122 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.547069    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7beb628-8d69-440b-8cee-b33026be56a0" containerName="revoke-keys"
Sep 16 12:32:40.547122 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.547079    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="118c0f39-5592-4165-92eb-4c594cb01941" containerName="payload-pre-processing"
Sep 16 12:32:40.549628 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.549614    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"
Sep 16 12:32:40.557645 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.557624    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:32:40.558424 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.558410    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:32:40.583587 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.583563    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"]
Sep 16 12:32:40.656183 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.656158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-352217-6cx4s\" (UID: \"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"
Sep 16 12:32:40.757377 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.757352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-352217-6cx4s\" (UID: \"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"
Sep 16 12:32:40.757976 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.757959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-352217-6cx4s\" (UID: \"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"
Sep 16 12:32:40.860288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:40.860263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"
Sep 16 12:32:41.146022 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:41.143242    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"]
Sep 16 12:32:41.152633 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:32:41.152603    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76c8ee9_8e13_4eeb_9ce3_67244b03b6d6.slice/crio-bfacf28c758d089ce571b44226a2f9e5199e75457ebba353eff9c6b68602f6ad WatchSource:0}: Error finding container bfacf28c758d089ce571b44226a2f9e5199e75457ebba353eff9c6b68602f6ad: Status 404 returned error can't find the container with id bfacf28c758d089ce571b44226a2f9e5199e75457ebba353eff9c6b68602f6ad
Sep 16 12:32:41.928858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:41.928828    2560 generic.go:356] "Generic (PLEG): container finished" podID="d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6" containerID="0e9dd481894fc1ea21a0ca3be937eadace12a140a626d43145687b29e2a95b94" exitCode=0
Sep 16 12:32:41.929180 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:41.928916    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s" event={"ID":"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6","Type":"ContainerDied","Data":"0e9dd481894fc1ea21a0ca3be937eadace12a140a626d43145687b29e2a95b94"}
Sep 16 12:32:41.929180 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:41.928973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s" event={"ID":"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6","Type":"ContainerStarted","Data":"bfacf28c758d089ce571b44226a2f9e5199e75457ebba353eff9c6b68602f6ad"}
Sep 16 12:32:43.069722 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:43.069698    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"
Sep 16 12:32:43.179387 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:43.179358    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6-maas-api-service-ca\") pod \"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6\" (UID: \"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6\") "
Sep 16 12:32:43.179662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:43.179642    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6" (UID: "d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:43.280065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:43.280007    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:43.942522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:43.942496    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"
Sep 16 12:32:43.942682 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:43.942492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s" event={"ID":"d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6","Type":"ContainerDied","Data":"bfacf28c758d089ce571b44226a2f9e5199e75457ebba353eff9c6b68602f6ad"}
Sep 16 12:32:43.942682 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:43.942611    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfacf28c758d089ce571b44226a2f9e5199e75457ebba353eff9c6b68602f6ad"
Sep 16 12:32:56.035090 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.035056    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"]
Sep 16 12:32:56.035486 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.035263    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z" podUID="777d3db6-9f85-4a46-a3ce-52b9b951dd2d" containerName="payload-processing" containerID="cri-o://547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0" gracePeriod=30
Sep 16 12:32:56.060886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.060859    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"]
Sep 16 12:32:56.061094 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.061070    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq" podUID="ec577a61-8a5a-42a6-a290-b3b84f6b93d5" containerName="payload-pre-processing" containerID="cri-o://b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5" gracePeriod=30
Sep 16 12:32:56.349500 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.349480    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:32:56.352538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.352520    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:32:56.492616 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.492588    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-plugins-config-volume\") pod \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") "
Sep 16 12:32:56.492766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.492625    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-plugins-config-volume\") pod \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") "
Sep 16 12:32:56.492766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.492657    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97kv9\" (UniqueName: \"kubernetes.io/projected/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-kube-api-access-97kv9\") pod \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\" (UID: \"777d3db6-9f85-4a46-a3ce-52b9b951dd2d\") "
Sep 16 12:32:56.492766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.492727    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq4f6\" (UniqueName: \"kubernetes.io/projected/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-kube-api-access-vq4f6\") pod \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\" (UID: \"ec577a61-8a5a-42a6-a290-b3b84f6b93d5\") "
Sep 16 12:32:56.493093 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.493064    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "777d3db6-9f85-4a46-a3ce-52b9b951dd2d" (UID: "777d3db6-9f85-4a46-a3ce-52b9b951dd2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:56.493093 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.493079    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec577a61-8a5a-42a6-a290-b3b84f6b93d5" (UID: "ec577a61-8a5a-42a6-a290-b3b84f6b93d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:56.495088 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.495060    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-kube-api-access-97kv9" (OuterVolumeSpecName: "kube-api-access-97kv9") pod "777d3db6-9f85-4a46-a3ce-52b9b951dd2d" (UID: "777d3db6-9f85-4a46-a3ce-52b9b951dd2d"). InnerVolumeSpecName "kube-api-access-97kv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:56.495162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.495103    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-kube-api-access-vq4f6" (OuterVolumeSpecName: "kube-api-access-vq4f6") pod "ec577a61-8a5a-42a6-a290-b3b84f6b93d5" (UID: "ec577a61-8a5a-42a6-a290-b3b84f6b93d5"). InnerVolumeSpecName "kube-api-access-vq4f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:56.515316 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.515294    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"]
Sep 16 12:32:56.529998 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.529977    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-352217-6cx4s"]
Sep 16 12:32:56.594182 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.594156    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:56.594182 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.594181    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:56.594317 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.594195    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kv9\" (UniqueName: \"kubernetes.io/projected/777d3db6-9f85-4a46-a3ce-52b9b951dd2d-kube-api-access-97kv9\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:56.594317 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:56.594207    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4f6\" (UniqueName: \"kubernetes.io/projected/ec577a61-8a5a-42a6-a290-b3b84f6b93d5-kube-api-access-vq4f6\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:32:57.020751 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.020667    2560 generic.go:356] "Generic (PLEG): container finished" podID="ec577a61-8a5a-42a6-a290-b3b84f6b93d5" containerID="b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5" exitCode=0
Sep 16 12:32:57.020751 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.020721    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"
Sep 16 12:32:57.020994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.020748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq" event={"ID":"ec577a61-8a5a-42a6-a290-b3b84f6b93d5","Type":"ContainerDied","Data":"b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5"}
Sep 16 12:32:57.020994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.020834    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq" event={"ID":"ec577a61-8a5a-42a6-a290-b3b84f6b93d5","Type":"ContainerDied","Data":"ac6c76e673e54ac1cc97e9ac33d763cacf963b1d1ceaf45b30d4d53774e9906b"}
Sep 16 12:32:57.020994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.020867    2560 scope.go:117] "RemoveContainer" containerID="b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5"
Sep 16 12:32:57.022285 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.022264    2560 generic.go:356] "Generic (PLEG): container finished" podID="777d3db6-9f85-4a46-a3ce-52b9b951dd2d" containerID="547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0" exitCode=0
Sep 16 12:32:57.022381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.022302    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"
Sep 16 12:32:57.022381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.022310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z" event={"ID":"777d3db6-9f85-4a46-a3ce-52b9b951dd2d","Type":"ContainerDied","Data":"547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0"}
Sep 16 12:32:57.022381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.022344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z" event={"ID":"777d3db6-9f85-4a46-a3ce-52b9b951dd2d","Type":"ContainerDied","Data":"2ba3634784be51ddf36447757104b3f689699b7b34e0910cf0bce80918c709a3"}
Sep 16 12:32:57.030965 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.030920    2560 scope.go:117] "RemoveContainer" containerID="b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5"
Sep 16 12:32:57.031341 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:32:57.031204    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5\": container with ID starting with b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5 not found: ID does not exist" containerID="b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5"
Sep 16 12:32:57.031341 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.031229    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5"} err="failed to get container status \"b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5\": rpc error: code = NotFound desc = could not find container \"b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5\": container with ID starting with b6922eed59d0b119f7712af6dec4b1c4348846c5f29a2cf04dc9767e029d06d5 not found: ID does not exist"
Sep 16 12:32:57.031341 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.031245    2560 scope.go:117] "RemoveContainer" containerID="547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0"
Sep 16 12:32:57.040224 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.040043    2560 scope.go:117] "RemoveContainer" containerID="547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0"
Sep 16 12:32:57.040453 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:32:57.040332    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0\": container with ID starting with 547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0 not found: ID does not exist" containerID="547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0"
Sep 16 12:32:57.040453 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.040353    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0"} err="failed to get container status \"547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0\": rpc error: code = NotFound desc = could not find container \"547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0\": container with ID starting with 547d3ed007a8799dd3d476272fb542e9e6e63324aa4a8566746062548cefcaf0 not found: ID does not exist"
Sep 16 12:32:57.064704 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.064675    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"]
Sep 16 12:32:57.072953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.072929    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-352217-576fdfb6fc-mf84z"]
Sep 16 12:32:57.097674 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.097655    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"]
Sep 16 12:32:57.105901 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:57.105880    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-352217-74cf89c58-qz9cq"]
Sep 16 12:32:58.325141 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:58.325100    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777d3db6-9f85-4a46-a3ce-52b9b951dd2d" path="/var/lib/kubelet/pods/777d3db6-9f85-4a46-a3ce-52b9b951dd2d/volumes"
Sep 16 12:32:58.325495 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:58.325428    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6" path="/var/lib/kubelet/pods/d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6/volumes"
Sep 16 12:32:58.325743 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:32:58.325730    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec577a61-8a5a-42a6-a290-b3b84f6b93d5" path="/var/lib/kubelet/pods/ec577a61-8a5a-42a6-a290-b3b84f6b93d5/volumes"
Sep 16 12:33:07.114273 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114240    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"]
Sep 16 12:33:07.114753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114644    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777d3db6-9f85-4a46-a3ce-52b9b951dd2d" containerName="payload-processing"
Sep 16 12:33:07.114753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114660    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="777d3db6-9f85-4a46-a3ce-52b9b951dd2d" containerName="payload-processing"
Sep 16 12:33:07.114753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114687    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6" containerName="revoke-keys"
Sep 16 12:33:07.114753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114697    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6" containerName="revoke-keys"
Sep 16 12:33:07.114753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114722    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec577a61-8a5a-42a6-a290-b3b84f6b93d5" containerName="payload-pre-processing"
Sep 16 12:33:07.114753 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114727    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec577a61-8a5a-42a6-a290-b3b84f6b93d5" containerName="payload-pre-processing"
Sep 16 12:33:07.115031 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114814    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d76c8ee9-8e13-4eeb-9ce3-67244b03b6d6" containerName="revoke-keys"
Sep 16 12:33:07.115031 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114825    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="777d3db6-9f85-4a46-a3ce-52b9b951dd2d" containerName="payload-processing"
Sep 16 12:33:07.115031 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.114835    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec577a61-8a5a-42a6-a290-b3b84f6b93d5" containerName="payload-pre-processing"
Sep 16 12:33:07.117993 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.117978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"
Sep 16 12:33:07.122098 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.122067    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:33:07.122098 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.122091    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:33:07.152407 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.152385    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"]
Sep 16 12:33:07.288924 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.288897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b649bf26-2de7-440b-b668-a23ac45289e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs\" (UID: \"b649bf26-2de7-440b-b668-a23ac45289e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"
Sep 16 12:33:07.389832 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.389726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b649bf26-2de7-440b-b668-a23ac45289e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs\" (UID: \"b649bf26-2de7-440b-b668-a23ac45289e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"
Sep 16 12:33:07.390329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.390310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b649bf26-2de7-440b-b668-a23ac45289e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs\" (UID: \"b649bf26-2de7-440b-b668-a23ac45289e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"
Sep 16 12:33:07.428493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.428471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"
Sep 16 12:33:07.705771 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:07.705705    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"]
Sep 16 12:33:07.709562 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:33:07.709538    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb649bf26_2de7_440b_b668_a23ac45289e0.slice/crio-338d6d9bc9fbfd840731ceeae456fb20d56b9697567197d3d70a7c4e50f8aee8 WatchSource:0}: Error finding container 338d6d9bc9fbfd840731ceeae456fb20d56b9697567197d3d70a7c4e50f8aee8: Status 404 returned error can't find the container with id 338d6d9bc9fbfd840731ceeae456fb20d56b9697567197d3d70a7c4e50f8aee8
Sep 16 12:33:08.091806 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.091755    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs" event={"ID":"b649bf26-2de7-440b-b668-a23ac45289e0","Type":"ContainerStarted","Data":"338d6d9bc9fbfd840731ceeae456fb20d56b9697567197d3d70a7c4e50f8aee8"}
Sep 16 12:33:08.228319 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.228292    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"]
Sep 16 12:33:08.231833 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.231803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"
Sep 16 12:33:08.264303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.264281    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"]
Sep 16 12:33:08.400502 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.400478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944e3460-ee67-4698-9559-2fc006381c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s\" (UID: \"944e3460-ee67-4698-9559-2fc006381c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"
Sep 16 12:33:08.501422 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.501396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944e3460-ee67-4698-9559-2fc006381c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s\" (UID: \"944e3460-ee67-4698-9559-2fc006381c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"
Sep 16 12:33:08.508312 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.502575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944e3460-ee67-4698-9559-2fc006381c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s\" (UID: \"944e3460-ee67-4698-9559-2fc006381c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"
Sep 16 12:33:08.542138 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.542112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"
Sep 16 12:33:08.825801 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:08.825707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"]
Sep 16 12:33:08.829927 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:33:08.829898    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944e3460_ee67_4698_9559_2fc006381c77.slice/crio-8e7721bce24a7494fabb14277828dd265f1db88a29e3c19c32b9f65b39e789f2 WatchSource:0}: Error finding container 8e7721bce24a7494fabb14277828dd265f1db88a29e3c19c32b9f65b39e789f2: Status 404 returned error can't find the container with id 8e7721bce24a7494fabb14277828dd265f1db88a29e3c19c32b9f65b39e789f2
Sep 16 12:33:09.098936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:09.098902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s" event={"ID":"944e3460-ee67-4698-9559-2fc006381c77","Type":"ContainerStarted","Data":"c1b681d861df3907b76b5fae46abb9aaa42a9da10848d1de413be0b2bf39b9ae"}
Sep 16 12:33:09.099055 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:09.098961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s" event={"ID":"944e3460-ee67-4698-9559-2fc006381c77","Type":"ContainerStarted","Data":"8e7721bce24a7494fabb14277828dd265f1db88a29e3c19c32b9f65b39e789f2"}
Sep 16 12:33:09.100664 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:09.100641    2560 generic.go:356] "Generic (PLEG): container finished" podID="b649bf26-2de7-440b-b668-a23ac45289e0" containerID="ad7bd095317c942bf7bb5378a02bb9ee7a2200d950491b6cce001fc2c37b7375" exitCode=0
Sep 16 12:33:09.100810 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:09.100682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs" event={"ID":"b649bf26-2de7-440b-b668-a23ac45289e0","Type":"ContainerDied","Data":"ad7bd095317c942bf7bb5378a02bb9ee7a2200d950491b6cce001fc2c37b7375"}
Sep 16 12:33:09.135841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:09.135777    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s" podStartSLOduration=1.135764547 podStartE2EDuration="1.135764547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:33:09.135669033 +0000 UTC m=+1785.270991003" watchObservedRunningTime="2026-09-16 12:33:09.135764547 +0000 UTC m=+1785.271086517"
Sep 16 12:33:10.109045 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:10.109011    2560 generic.go:356] "Generic (PLEG): container finished" podID="944e3460-ee67-4698-9559-2fc006381c77" containerID="c1b681d861df3907b76b5fae46abb9aaa42a9da10848d1de413be0b2bf39b9ae" exitCode=0
Sep 16 12:33:10.109471 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:10.109104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s" event={"ID":"944e3460-ee67-4698-9559-2fc006381c77","Type":"ContainerDied","Data":"c1b681d861df3907b76b5fae46abb9aaa42a9da10848d1de413be0b2bf39b9ae"}
Sep 16 12:33:10.241827 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:10.241800    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"
Sep 16 12:33:10.420191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:10.420131    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b649bf26-2de7-440b-b668-a23ac45289e0-maas-api-service-ca\") pod \"b649bf26-2de7-440b-b668-a23ac45289e0\" (UID: \"b649bf26-2de7-440b-b668-a23ac45289e0\") "
Sep 16 12:33:10.420761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:10.420493    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b649bf26-2de7-440b-b668-a23ac45289e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b649bf26-2de7-440b-b668-a23ac45289e0" (UID: "b649bf26-2de7-440b-b668-a23ac45289e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:10.420761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:10.420562    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b649bf26-2de7-440b-b668-a23ac45289e0-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:11.117642 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:11.117611    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"
Sep 16 12:33:11.118042 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:11.117647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs" event={"ID":"b649bf26-2de7-440b-b668-a23ac45289e0","Type":"ContainerDied","Data":"338d6d9bc9fbfd840731ceeae456fb20d56b9697567197d3d70a7c4e50f8aee8"}
Sep 16 12:33:11.118042 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:11.117696    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="338d6d9bc9fbfd840731ceeae456fb20d56b9697567197d3d70a7c4e50f8aee8"
Sep 16 12:33:11.257885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:11.257861    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"
Sep 16 12:33:11.430201 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:11.430126    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944e3460-ee67-4698-9559-2fc006381c77-maas-api-service-ca\") pod \"944e3460-ee67-4698-9559-2fc006381c77\" (UID: \"944e3460-ee67-4698-9559-2fc006381c77\") "
Sep 16 12:33:11.430476 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:11.430454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944e3460-ee67-4698-9559-2fc006381c77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "944e3460-ee67-4698-9559-2fc006381c77" (UID: "944e3460-ee67-4698-9559-2fc006381c77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:11.531298 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:11.531273    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944e3460-ee67-4698-9559-2fc006381c77-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:12.125281 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:12.125253    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"
Sep 16 12:33:12.125636 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:12.125253    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s" event={"ID":"944e3460-ee67-4698-9559-2fc006381c77","Type":"ContainerDied","Data":"8e7721bce24a7494fabb14277828dd265f1db88a29e3c19c32b9f65b39e789f2"}
Sep 16 12:33:12.125636 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:12.125392    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7721bce24a7494fabb14277828dd265f1db88a29e3c19c32b9f65b39e789f2"
Sep 16 12:33:13.122867 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:13.122835    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"]
Sep 16 12:33:13.179475 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:13.179449    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"]
Sep 16 12:33:13.192917 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:13.192894    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"]
Sep 16 12:33:18.107000 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:18.106910    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"]
Sep 16 12:33:18.109347 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:18.109320    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cadd00-29826030-kh4fx"]
Sep 16 12:33:18.324889 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:18.324856    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1" path="/var/lib/kubelet/pods/d60f9cbc-0bd3-48d1-ab1e-a7413ee102d1/volumes"
Sep 16 12:33:18.642548 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:18.642508    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"]
Sep 16 12:33:18.658838 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:18.658812    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cadd00-c99bs"]
Sep 16 12:33:20.325729 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:20.325691    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b649bf26-2de7-440b-b668-a23ac45289e0" path="/var/lib/kubelet/pods/b649bf26-2de7-440b-b668-a23ac45289e0/volumes"
Sep 16 12:33:23.135237 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.135203    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"]
Sep 16 12:33:23.148164 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.148135    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-68f7cd-29826030-5xmpl"]
Sep 16 12:33:23.230370 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.230341    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"]
Sep 16 12:33:23.230568 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.230540    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q" podUID="ce24f8c7-fe43-45cd-add7-74ab52821417" containerName="payload-processing" containerID="cri-o://ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3" gracePeriod=30
Sep 16 12:33:23.275827 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.275781    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"]
Sep 16 12:33:23.276064 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.276036    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj" podUID="51be8bcf-f26b-4f65-901e-f4b16921995c" containerName="payload-pre-processing" containerID="cri-o://d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd" gracePeriod=30
Sep 16 12:33:23.525139 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.525113    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:33:23.554447 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.554427    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:33:23.635425 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.635396    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhdvm\" (UniqueName: \"kubernetes.io/projected/ce24f8c7-fe43-45cd-add7-74ab52821417-kube-api-access-xhdvm\") pod \"ce24f8c7-fe43-45cd-add7-74ab52821417\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") "
Sep 16 12:33:23.635582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.635429    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttmmk\" (UniqueName: \"kubernetes.io/projected/51be8bcf-f26b-4f65-901e-f4b16921995c-kube-api-access-ttmmk\") pod \"51be8bcf-f26b-4f65-901e-f4b16921995c\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") "
Sep 16 12:33:23.635582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.635477    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce24f8c7-fe43-45cd-add7-74ab52821417-plugins-config-volume\") pod \"ce24f8c7-fe43-45cd-add7-74ab52821417\" (UID: \"ce24f8c7-fe43-45cd-add7-74ab52821417\") "
Sep 16 12:33:23.635582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.635509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51be8bcf-f26b-4f65-901e-f4b16921995c-plugins-config-volume\") pod \"51be8bcf-f26b-4f65-901e-f4b16921995c\" (UID: \"51be8bcf-f26b-4f65-901e-f4b16921995c\") "
Sep 16 12:33:23.635933 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.635907    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce24f8c7-fe43-45cd-add7-74ab52821417-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce24f8c7-fe43-45cd-add7-74ab52821417" (UID: "ce24f8c7-fe43-45cd-add7-74ab52821417"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:23.636048 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.635954    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51be8bcf-f26b-4f65-901e-f4b16921995c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51be8bcf-f26b-4f65-901e-f4b16921995c" (UID: "51be8bcf-f26b-4f65-901e-f4b16921995c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:23.637657 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.637637    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce24f8c7-fe43-45cd-add7-74ab52821417-kube-api-access-xhdvm" (OuterVolumeSpecName: "kube-api-access-xhdvm") pod "ce24f8c7-fe43-45cd-add7-74ab52821417" (UID: "ce24f8c7-fe43-45cd-add7-74ab52821417"). InnerVolumeSpecName "kube-api-access-xhdvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:23.637754 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.637740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51be8bcf-f26b-4f65-901e-f4b16921995c-kube-api-access-ttmmk" (OuterVolumeSpecName: "kube-api-access-ttmmk") pod "51be8bcf-f26b-4f65-901e-f4b16921995c" (UID: "51be8bcf-f26b-4f65-901e-f4b16921995c"). InnerVolumeSpecName "kube-api-access-ttmmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:23.737117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.737055    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce24f8c7-fe43-45cd-add7-74ab52821417-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:23.737117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.737079    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51be8bcf-f26b-4f65-901e-f4b16921995c-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:23.737117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.737089    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhdvm\" (UniqueName: \"kubernetes.io/projected/ce24f8c7-fe43-45cd-add7-74ab52821417-kube-api-access-xhdvm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:23.737117 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:23.737098    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttmmk\" (UniqueName: \"kubernetes.io/projected/51be8bcf-f26b-4f65-901e-f4b16921995c-kube-api-access-ttmmk\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:24.037141 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.037065    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"]
Sep 16 12:33:24.050982 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.050951    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-68f7cd-8rh7s"]
Sep 16 12:33:24.200003 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.199967    2560 generic.go:356] "Generic (PLEG): container finished" podID="51be8bcf-f26b-4f65-901e-f4b16921995c" containerID="d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd" exitCode=0
Sep 16 12:33:24.200420 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.200021    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"
Sep 16 12:33:24.200420 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.200048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj" event={"ID":"51be8bcf-f26b-4f65-901e-f4b16921995c","Type":"ContainerDied","Data":"d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd"}
Sep 16 12:33:24.200420 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.200135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj" event={"ID":"51be8bcf-f26b-4f65-901e-f4b16921995c","Type":"ContainerDied","Data":"d0e95caa001cc5313c298ebb2cf89df8f22579d8f7781b69a1e7e61d618b853e"}
Sep 16 12:33:24.200420 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.200163    2560 scope.go:117] "RemoveContainer" containerID="d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd"
Sep 16 12:33:24.201548 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.201523    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce24f8c7-fe43-45cd-add7-74ab52821417" containerID="ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3" exitCode=0
Sep 16 12:33:24.201629 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.201550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q" event={"ID":"ce24f8c7-fe43-45cd-add7-74ab52821417","Type":"ContainerDied","Data":"ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3"}
Sep 16 12:33:24.201629 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.201585    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"
Sep 16 12:33:24.201629 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.201602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q" event={"ID":"ce24f8c7-fe43-45cd-add7-74ab52821417","Type":"ContainerDied","Data":"e8667480b04353c50ff2e45bebc513d9d427c389d1b0c5aebd2618a89d851f9b"}
Sep 16 12:33:24.211111 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.211092    2560 scope.go:117] "RemoveContainer" containerID="d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd"
Sep 16 12:33:24.211447 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:33:24.211420    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd\": container with ID starting with d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd not found: ID does not exist" containerID="d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd"
Sep 16 12:33:24.211538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.211458    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd"} err="failed to get container status \"d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd\": rpc error: code = NotFound desc = could not find container \"d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd\": container with ID starting with d979ce20241e717396faf4f3074a6a788523890f898d520f64bb1faf37877abd not found: ID does not exist"
Sep 16 12:33:24.211538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.211481    2560 scope.go:117] "RemoveContainer" containerID="ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3"
Sep 16 12:33:24.221008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.220993    2560 scope.go:117] "RemoveContainer" containerID="ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3"
Sep 16 12:33:24.221261 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:33:24.221241    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3\": container with ID starting with ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3 not found: ID does not exist" containerID="ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3"
Sep 16 12:33:24.221321 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.221271    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3"} err="failed to get container status \"ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3\": rpc error: code = NotFound desc = could not find container \"ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3\": container with ID starting with ec0644a88f9414677a5a62d304a637eedf33beee0dd6209c229e7f376fe366b3 not found: ID does not exist"
Sep 16 12:33:24.253726 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.253704    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"]
Sep 16 12:33:24.271410 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.271391    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-68f7cd-6cbd566bc4-5vpfj"]
Sep 16 12:33:24.325010 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.324960    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51be8bcf-f26b-4f65-901e-f4b16921995c" path="/var/lib/kubelet/pods/51be8bcf-f26b-4f65-901e-f4b16921995c/volumes"
Sep 16 12:33:24.325273 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.325260    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="944e3460-ee67-4698-9559-2fc006381c77" path="/var/lib/kubelet/pods/944e3460-ee67-4698-9559-2fc006381c77/volumes"
Sep 16 12:33:24.325501 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.325490    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a84184f5-80a9-4ff5-8791-2d9f4cff0ff4" path="/var/lib/kubelet/pods/a84184f5-80a9-4ff5-8791-2d9f4cff0ff4/volumes"
Sep 16 12:33:24.357949 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.357929    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"]
Sep 16 12:33:24.368261 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.368239    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-68f7cd-b87c4cd47-hkq6q"]
Sep 16 12:33:24.424829 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.424805    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:33:24.429936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.429911    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:33:24.621320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:24.621282    2560 scope.go:117] "RemoveContainer" containerID="9fbae2795e397b4d420b7627e9473d48f294a4c57388fc8f5163c17c3b71d83e"
Sep 16 12:33:26.324990 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:26.324954    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce24f8c7-fe43-45cd-add7-74ab52821417" path="/var/lib/kubelet/pods/ce24f8c7-fe43-45cd-add7-74ab52821417/volumes"
Sep 16 12:33:28.142713 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.142674    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk" podUID="73efacf5-21cc-4b18-8c6e-446d77915866" containerName="main" containerID="cri-o://97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb" gracePeriod=45
Sep 16 12:33:28.197871 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.197842    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4" podUID="b3aa3376-682f-4569-ada5-bd79f40d5c88" containerName="main" containerID="cri-o://88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb" gracePeriod=45
Sep 16 12:33:28.211150 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.211116    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc" podUID="803743e9-b4e7-4d18-986d-f1f221fd2524" containerName="main" containerID="cri-o://4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6" gracePeriod=45
Sep 16 12:33:28.485997 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.485969    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:33:28.496730 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.496703    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:33:28.504472 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.504451    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:33:28.585147 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585120    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-model-cache\") pod \"803743e9-b4e7-4d18-986d-f1f221fd2524\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") "
Sep 16 12:33:28.585147 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585151    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-home\") pod \"b3aa3376-682f-4569-ada5-bd79f40d5c88\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") "
Sep 16 12:33:28.585364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585182    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vvjb\" (UniqueName: \"kubernetes.io/projected/73efacf5-21cc-4b18-8c6e-446d77915866-kube-api-access-4vvjb\") pod \"73efacf5-21cc-4b18-8c6e-446d77915866\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") "
Sep 16 12:33:28.585364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585201    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f47r\" (UniqueName: \"kubernetes.io/projected/b3aa3376-682f-4569-ada5-bd79f40d5c88-kube-api-access-7f47r\") pod \"b3aa3376-682f-4569-ada5-bd79f40d5c88\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") "
Sep 16 12:33:28.585364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-tmp-dir\") pod \"73efacf5-21cc-4b18-8c6e-446d77915866\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") "
Sep 16 12:33:28.585364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-dshm\") pod \"803743e9-b4e7-4d18-986d-f1f221fd2524\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") "
Sep 16 12:33:28.585579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585480    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-home" (OuterVolumeSpecName: "home") pod "b3aa3376-682f-4569-ada5-bd79f40d5c88" (UID: "b3aa3376-682f-4569-ada5-bd79f40d5c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.585579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-model-cache" (OuterVolumeSpecName: "model-cache") pod "803743e9-b4e7-4d18-986d-f1f221fd2524" (UID: "803743e9-b4e7-4d18-986d-f1f221fd2524"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.585579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585508    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73efacf5-21cc-4b18-8c6e-446d77915866" (UID: "73efacf5-21cc-4b18-8c6e-446d77915866"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.585579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585568    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-model-cache\") pod \"b3aa3376-682f-4569-ada5-bd79f40d5c88\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") "
Sep 16 12:33:28.585779 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585638    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73efacf5-21cc-4b18-8c6e-446d77915866-tls-certs\") pod \"73efacf5-21cc-4b18-8c6e-446d77915866\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") "
Sep 16 12:33:28.585779 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585683    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803743e9-b4e7-4d18-986d-f1f221fd2524-tls-certs\") pod \"803743e9-b4e7-4d18-986d-f1f221fd2524\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") "
Sep 16 12:33:28.585779 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.585763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3aa3376-682f-4569-ada5-bd79f40d5c88" (UID: "b3aa3376-682f-4569-ada5-bd79f40d5c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586015    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-tmp-dir\") pod \"b3aa3376-682f-4569-ada5-bd79f40d5c88\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586078    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-home\") pod \"73efacf5-21cc-4b18-8c6e-446d77915866\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586109    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-tmp-dir\") pod \"803743e9-b4e7-4d18-986d-f1f221fd2524\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586159    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-dshm\") pod \"b3aa3376-682f-4569-ada5-bd79f40d5c88\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586207    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3aa3376-682f-4569-ada5-bd79f40d5c88-tls-certs\") pod \"b3aa3376-682f-4569-ada5-bd79f40d5c88\" (UID: \"b3aa3376-682f-4569-ada5-bd79f40d5c88\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586231    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-dshm\") pod \"73efacf5-21cc-4b18-8c6e-446d77915866\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3aa3376-682f-4569-ada5-bd79f40d5c88" (UID: "b3aa3376-682f-4569-ada5-bd79f40d5c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lk7s\" (UniqueName: \"kubernetes.io/projected/803743e9-b4e7-4d18-986d-f1f221fd2524-kube-api-access-9lk7s\") pod \"803743e9-b4e7-4d18-986d-f1f221fd2524\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586317    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-model-cache\") pod \"73efacf5-21cc-4b18-8c6e-446d77915866\" (UID: \"73efacf5-21cc-4b18-8c6e-446d77915866\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-home\") pod \"803743e9-b4e7-4d18-986d-f1f221fd2524\" (UID: \"803743e9-b4e7-4d18-986d-f1f221fd2524\") "
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586387    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-home" (OuterVolumeSpecName: "home") pod "73efacf5-21cc-4b18-8c6e-446d77915866" (UID: "73efacf5-21cc-4b18-8c6e-446d77915866"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586583    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-model-cache" (OuterVolumeSpecName: "model-cache") pod "73efacf5-21cc-4b18-8c6e-446d77915866" (UID: "73efacf5-21cc-4b18-8c6e-446d77915866"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "803743e9-b4e7-4d18-986d-f1f221fd2524" (UID: "803743e9-b4e7-4d18-986d-f1f221fd2524"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.586814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-home" (OuterVolumeSpecName: "home") pod "803743e9-b4e7-4d18-986d-f1f221fd2524" (UID: "803743e9-b4e7-4d18-986d-f1f221fd2524"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.587721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586844    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.587721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586866    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.587721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586879    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.587721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586892    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.587721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586903    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.587721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586915    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.587721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.586928    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.588302 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.588179    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aa3376-682f-4569-ada5-bd79f40d5c88-kube-api-access-7f47r" (OuterVolumeSpecName: "kube-api-access-7f47r") pod "b3aa3376-682f-4569-ada5-bd79f40d5c88" (UID: "b3aa3376-682f-4569-ada5-bd79f40d5c88"). InnerVolumeSpecName "kube-api-access-7f47r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:28.588850 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.588814    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73efacf5-21cc-4b18-8c6e-446d77915866-kube-api-access-4vvjb" (OuterVolumeSpecName: "kube-api-access-4vvjb") pod "73efacf5-21cc-4b18-8c6e-446d77915866" (UID: "73efacf5-21cc-4b18-8c6e-446d77915866"). InnerVolumeSpecName "kube-api-access-4vvjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:28.589403 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.589371    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-dshm" (OuterVolumeSpecName: "dshm") pod "803743e9-b4e7-4d18-986d-f1f221fd2524" (UID: "803743e9-b4e7-4d18-986d-f1f221fd2524"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.589403 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.589374    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3aa3376-682f-4569-ada5-bd79f40d5c88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3aa3376-682f-4569-ada5-bd79f40d5c88" (UID: "b3aa3376-682f-4569-ada5-bd79f40d5c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:28.589577 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.589532    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803743e9-b4e7-4d18-986d-f1f221fd2524-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "803743e9-b4e7-4d18-986d-f1f221fd2524" (UID: "803743e9-b4e7-4d18-986d-f1f221fd2524"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:28.589820 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.589766    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-dshm" (OuterVolumeSpecName: "dshm") pod "73efacf5-21cc-4b18-8c6e-446d77915866" (UID: "73efacf5-21cc-4b18-8c6e-446d77915866"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.590220 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.590200    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-dshm" (OuterVolumeSpecName: "dshm") pod "b3aa3376-682f-4569-ada5-bd79f40d5c88" (UID: "b3aa3376-682f-4569-ada5-bd79f40d5c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:33:28.590296 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.590250    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803743e9-b4e7-4d18-986d-f1f221fd2524-kube-api-access-9lk7s" (OuterVolumeSpecName: "kube-api-access-9lk7s") pod "803743e9-b4e7-4d18-986d-f1f221fd2524" (UID: "803743e9-b4e7-4d18-986d-f1f221fd2524"). InnerVolumeSpecName "kube-api-access-9lk7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:28.590625 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.590609    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73efacf5-21cc-4b18-8c6e-446d77915866-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73efacf5-21cc-4b18-8c6e-446d77915866" (UID: "73efacf5-21cc-4b18-8c6e-446d77915866"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:28.688329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688262    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f47r\" (UniqueName: \"kubernetes.io/projected/b3aa3376-682f-4569-ada5-bd79f40d5c88-kube-api-access-7f47r\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688288    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688298    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73efacf5-21cc-4b18-8c6e-446d77915866-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688307    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803743e9-b4e7-4d18-986d-f1f221fd2524-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688314    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688321    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3aa3376-682f-4569-ada5-bd79f40d5c88-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688328    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3aa3376-682f-4569-ada5-bd79f40d5c88-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688336    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73efacf5-21cc-4b18-8c6e-446d77915866-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688343    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lk7s\" (UniqueName: \"kubernetes.io/projected/803743e9-b4e7-4d18-986d-f1f221fd2524-kube-api-access-9lk7s\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688351    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803743e9-b4e7-4d18-986d-f1f221fd2524-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:28.688632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:28.688359    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvjb\" (UniqueName: \"kubernetes.io/projected/73efacf5-21cc-4b18-8c6e-446d77915866-kube-api-access-4vvjb\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:29.235754 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.235718    2560 generic.go:356] "Generic (PLEG): container finished" podID="b3aa3376-682f-4569-ada5-bd79f40d5c88" containerID="88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb" exitCode=0
Sep 16 12:33:29.236195 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.235761    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"
Sep 16 12:33:29.236195 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.235810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4" event={"ID":"b3aa3376-682f-4569-ada5-bd79f40d5c88","Type":"ContainerDied","Data":"88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb"}
Sep 16 12:33:29.236195 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.235852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4" event={"ID":"b3aa3376-682f-4569-ada5-bd79f40d5c88","Type":"ContainerDied","Data":"8ce3a2a990463c346722904bfef53f9261c3cf1bccae7a6c04c408c35a5812aa"}
Sep 16 12:33:29.236195 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.235872    2560 scope.go:117] "RemoveContainer" containerID="88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb"
Sep 16 12:33:29.237308 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.237276    2560 generic.go:356] "Generic (PLEG): container finished" podID="73efacf5-21cc-4b18-8c6e-446d77915866" containerID="97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb" exitCode=0
Sep 16 12:33:29.237428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.237324    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"
Sep 16 12:33:29.237428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.237371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk" event={"ID":"73efacf5-21cc-4b18-8c6e-446d77915866","Type":"ContainerDied","Data":"97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb"}
Sep 16 12:33:29.237428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.237418    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk" event={"ID":"73efacf5-21cc-4b18-8c6e-446d77915866","Type":"ContainerDied","Data":"477266280aa7c24a580acaeb8e847e1f3e43ec5fa06e84ef9e20e1f30e66def3"}
Sep 16 12:33:29.239027 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.239003    2560 generic.go:356] "Generic (PLEG): container finished" podID="803743e9-b4e7-4d18-986d-f1f221fd2524" containerID="4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6" exitCode=0
Sep 16 12:33:29.239152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.239034    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"
Sep 16 12:33:29.239152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.239089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc" event={"ID":"803743e9-b4e7-4d18-986d-f1f221fd2524","Type":"ContainerDied","Data":"4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6"}
Sep 16 12:33:29.239152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.239138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc" event={"ID":"803743e9-b4e7-4d18-986d-f1f221fd2524","Type":"ContainerDied","Data":"16e870d33daf194ce18c0e1fa097415cfc497c246f6418bee6d68e3bdc88c64f"}
Sep 16 12:33:29.246714 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.246696    2560 scope.go:117] "RemoveContainer" containerID="88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb"
Sep 16 12:33:29.247150 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:33:29.247050    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb\": container with ID starting with 88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb not found: ID does not exist" containerID="88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb"
Sep 16 12:33:29.247150 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.247087    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb"} err="failed to get container status \"88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb\": rpc error: code = NotFound desc = could not find container \"88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb\": container with ID starting with 88930dad8fb31afa38ba09fcf9bfc577666d5f329cf36cc0901f3434cae250cb not found: ID does not exist"
Sep 16 12:33:29.247150 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.247109    2560 scope.go:117] "RemoveContainer" containerID="97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb"
Sep 16 12:33:29.255188 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.255164    2560 scope.go:117] "RemoveContainer" containerID="97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb"
Sep 16 12:33:29.255439 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:33:29.255419    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb\": container with ID starting with 97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb not found: ID does not exist" containerID="97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb"
Sep 16 12:33:29.255522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.255449    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb"} err="failed to get container status \"97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb\": rpc error: code = NotFound desc = could not find container \"97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb\": container with ID starting with 97968fd07234061184ba78fb3b9ad2b1cd2cc9fd677c2495ab95ca44d868cbcb not found: ID does not exist"
Sep 16 12:33:29.255522 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.255478    2560 scope.go:117] "RemoveContainer" containerID="4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6"
Sep 16 12:33:29.262943 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.262927    2560 scope.go:117] "RemoveContainer" containerID="4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6"
Sep 16 12:33:29.263201 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:33:29.263184    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6\": container with ID starting with 4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6 not found: ID does not exist" containerID="4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6"
Sep 16 12:33:29.263255 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.263206    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6"} err="failed to get container status \"4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6\": rpc error: code = NotFound desc = could not find container \"4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6\": container with ID starting with 4d80ef9a5d4bd63310c64cf4cbe7c72229719fbc21d6dc75ce616d2a8c7b72b6 not found: ID does not exist"
Sep 16 12:33:29.371131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.371105    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"]
Sep 16 12:33:29.382660 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.382637    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/distinct-2-w4-68f7cd-kserve-78955d7958-trnxk"]
Sep 16 12:33:29.434610 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.434558    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"]
Sep 16 12:33:29.494019 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.493995    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/simulator-w4-68f7cd-kserve-7c5c5fbdbc-fxxcc"]
Sep 16 12:33:29.534706 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.534683    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"]
Sep 16 12:33:29.568291 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:29.568269    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-68f7cd/premium-w4-68f7cd-kserve-b46775474-b4jc4"]
Sep 16 12:33:30.325277 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:30.325236    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73efacf5-21cc-4b18-8c6e-446d77915866" path="/var/lib/kubelet/pods/73efacf5-21cc-4b18-8c6e-446d77915866/volumes"
Sep 16 12:33:30.325801 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:30.325761    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="803743e9-b4e7-4d18-986d-f1f221fd2524" path="/var/lib/kubelet/pods/803743e9-b4e7-4d18-986d-f1f221fd2524/volumes"
Sep 16 12:33:30.326193 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:30.326174    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3aa3376-682f-4569-ada5-bd79f40d5c88" path="/var/lib/kubelet/pods/b3aa3376-682f-4569-ada5-bd79f40d5c88/volumes"
Sep 16 12:33:34.075919 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.075888    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"]
Sep 16 12:33:34.076513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076493    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b649bf26-2de7-440b-b668-a23ac45289e0" containerName="revoke-keys"
Sep 16 12:33:34.076597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076517    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b649bf26-2de7-440b-b668-a23ac45289e0" containerName="revoke-keys"
Sep 16 12:33:34.076597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076533    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73efacf5-21cc-4b18-8c6e-446d77915866" containerName="main"
Sep 16 12:33:34.076597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076541    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="73efacf5-21cc-4b18-8c6e-446d77915866" containerName="main"
Sep 16 12:33:34.076597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076569    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3aa3376-682f-4569-ada5-bd79f40d5c88" containerName="main"
Sep 16 12:33:34.076597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076579    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3aa3376-682f-4569-ada5-bd79f40d5c88" containerName="main"
Sep 16 12:33:34.076597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076589    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944e3460-ee67-4698-9559-2fc006381c77" containerName="revoke-keys"
Sep 16 12:33:34.076597 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076597    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="944e3460-ee67-4698-9559-2fc006381c77" containerName="revoke-keys"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076611    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51be8bcf-f26b-4f65-901e-f4b16921995c" containerName="payload-pre-processing"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076620    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="51be8bcf-f26b-4f65-901e-f4b16921995c" containerName="payload-pre-processing"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076636    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce24f8c7-fe43-45cd-add7-74ab52821417" containerName="payload-processing"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076646    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce24f8c7-fe43-45cd-add7-74ab52821417" containerName="payload-processing"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076661    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="803743e9-b4e7-4d18-986d-f1f221fd2524" containerName="main"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076668    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="803743e9-b4e7-4d18-986d-f1f221fd2524" containerName="main"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076761    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="51be8bcf-f26b-4f65-901e-f4b16921995c" containerName="payload-pre-processing"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076772    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="803743e9-b4e7-4d18-986d-f1f221fd2524" containerName="main"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076808    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce24f8c7-fe43-45cd-add7-74ab52821417" containerName="payload-processing"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076821    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3aa3376-682f-4569-ada5-bd79f40d5c88" containerName="main"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076830    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="73efacf5-21cc-4b18-8c6e-446d77915866" containerName="main"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076839    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b649bf26-2de7-440b-b668-a23ac45289e0" containerName="revoke-keys"
Sep 16 12:33:34.076986 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.076847    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="944e3460-ee67-4698-9559-2fc006381c77" containerName="revoke-keys"
Sep 16 12:33:34.081369 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.081350    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"
Sep 16 12:33:34.110910 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.110888    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:33:34.111068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.110891    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:33:34.134658 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.134634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53874ddb-6613-4e6f-879d-1461c47d06e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft\" (UID: \"53874ddb-6613-4e6f-879d-1461c47d06e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"
Sep 16 12:33:34.149872 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.149848    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"]
Sep 16 12:33:34.235591 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.235559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53874ddb-6613-4e6f-879d-1461c47d06e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft\" (UID: \"53874ddb-6613-4e6f-879d-1461c47d06e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"
Sep 16 12:33:34.236159 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.236139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53874ddb-6613-4e6f-879d-1461c47d06e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft\" (UID: \"53874ddb-6613-4e6f-879d-1461c47d06e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"
Sep 16 12:33:34.390887 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.390867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"
Sep 16 12:33:34.681944 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:34.681922    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"]
Sep 16 12:33:34.684997 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:33:34.684970    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53874ddb_6613_4e6f_879d_1461c47d06e9.slice/crio-dbbe166d311169e63d1ba08141f6a70ac34c8889b28f293e9682403224a441e8 WatchSource:0}: Error finding container dbbe166d311169e63d1ba08141f6a70ac34c8889b28f293e9682403224a441e8: Status 404 returned error can't find the container with id dbbe166d311169e63d1ba08141f6a70ac34c8889b28f293e9682403224a441e8
Sep 16 12:33:35.277301 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:35.277226    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft" event={"ID":"53874ddb-6613-4e6f-879d-1461c47d06e9","Type":"ContainerStarted","Data":"140e0d15376a8e9b689ef43cc716d11fd29e66036bd63b7c6e57ebc20ed1b7bc"}
Sep 16 12:33:35.277301 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:35.277281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft" event={"ID":"53874ddb-6613-4e6f-879d-1461c47d06e9","Type":"ContainerStarted","Data":"dbbe166d311169e63d1ba08141f6a70ac34c8889b28f293e9682403224a441e8"}
Sep 16 12:33:35.339355 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:35.339311    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft" podStartSLOduration=1.339298339 podStartE2EDuration="1.339298339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:33:35.336779764 +0000 UTC m=+1811.472101735" watchObservedRunningTime="2026-09-16 12:33:35.339298339 +0000 UTC m=+1811.474620307"
Sep 16 12:33:36.283940 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:36.283907    2560 generic.go:356] "Generic (PLEG): container finished" podID="53874ddb-6613-4e6f-879d-1461c47d06e9" containerID="140e0d15376a8e9b689ef43cc716d11fd29e66036bd63b7c6e57ebc20ed1b7bc" exitCode=0
Sep 16 12:33:36.284340 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:36.283996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft" event={"ID":"53874ddb-6613-4e6f-879d-1461c47d06e9","Type":"ContainerDied","Data":"140e0d15376a8e9b689ef43cc716d11fd29e66036bd63b7c6e57ebc20ed1b7bc"}
Sep 16 12:33:37.443267 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:37.443239    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"
Sep 16 12:33:37.563972 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:37.563887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53874ddb-6613-4e6f-879d-1461c47d06e9-maas-api-service-ca\") pod \"53874ddb-6613-4e6f-879d-1461c47d06e9\" (UID: \"53874ddb-6613-4e6f-879d-1461c47d06e9\") "
Sep 16 12:33:37.564197 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:37.564174    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53874ddb-6613-4e6f-879d-1461c47d06e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53874ddb-6613-4e6f-879d-1461c47d06e9" (UID: "53874ddb-6613-4e6f-879d-1461c47d06e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:37.664757 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:37.664718    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53874ddb-6613-4e6f-879d-1461c47d06e9-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:38.299723 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:38.299692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft" event={"ID":"53874ddb-6613-4e6f-879d-1461c47d06e9","Type":"ContainerDied","Data":"dbbe166d311169e63d1ba08141f6a70ac34c8889b28f293e9682403224a441e8"}
Sep 16 12:33:38.299896 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:38.299744    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbe166d311169e63d1ba08141f6a70ac34c8889b28f293e9682403224a441e8"
Sep 16 12:33:38.299896 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:38.299700    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"
Sep 16 12:33:45.245872 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.245836    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"]
Sep 16 12:33:45.251538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.251513    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-51ae50-29826030-bd5p5"]
Sep 16 12:33:45.336623 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.336592    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"]
Sep 16 12:33:45.336837 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.336807    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf" podUID="e5cac034-1725-41a5-ad7d-690671b7d6d8" containerName="payload-pre-processing" containerID="cri-o://6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77" gracePeriod=30
Sep 16 12:33:45.595308 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.595287    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:33:45.738199 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.738174    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cac034-1725-41a5-ad7d-690671b7d6d8-plugins-config-volume\") pod \"e5cac034-1725-41a5-ad7d-690671b7d6d8\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") "
Sep 16 12:33:45.738349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.738236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56s4w\" (UniqueName: \"kubernetes.io/projected/e5cac034-1725-41a5-ad7d-690671b7d6d8-kube-api-access-56s4w\") pod \"e5cac034-1725-41a5-ad7d-690671b7d6d8\" (UID: \"e5cac034-1725-41a5-ad7d-690671b7d6d8\") "
Sep 16 12:33:45.738540 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.738511    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cac034-1725-41a5-ad7d-690671b7d6d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5cac034-1725-41a5-ad7d-690671b7d6d8" (UID: "e5cac034-1725-41a5-ad7d-690671b7d6d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:45.740463 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.740438    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cac034-1725-41a5-ad7d-690671b7d6d8-kube-api-access-56s4w" (OuterVolumeSpecName: "kube-api-access-56s4w") pod "e5cac034-1725-41a5-ad7d-690671b7d6d8" (UID: "e5cac034-1725-41a5-ad7d-690671b7d6d8"). InnerVolumeSpecName "kube-api-access-56s4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:45.832442 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.832387    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"]
Sep 16 12:33:45.839232 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.839206    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cac034-1725-41a5-ad7d-690671b7d6d8-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.839232 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.839234    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56s4w\" (UniqueName: \"kubernetes.io/projected/e5cac034-1725-41a5-ad7d-690671b7d6d8-kube-api-access-56s4w\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.839883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:45.839866    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-51ae50-9mfft"]
Sep 16 12:33:46.325546 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.325512    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291d3def-3e9e-4e0b-8065-79182a3cb759" path="/var/lib/kubelet/pods/291d3def-3e9e-4e0b-8065-79182a3cb759/volumes"
Sep 16 12:33:46.325917 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.325817    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53874ddb-6613-4e6f-879d-1461c47d06e9" path="/var/lib/kubelet/pods/53874ddb-6613-4e6f-879d-1461c47d06e9/volumes"
Sep 16 12:33:46.352667 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.352643    2560 generic.go:356] "Generic (PLEG): container finished" podID="e5cac034-1725-41a5-ad7d-690671b7d6d8" containerID="6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77" exitCode=0
Sep 16 12:33:46.352778 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.352682    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"
Sep 16 12:33:46.352778 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.352704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf" event={"ID":"e5cac034-1725-41a5-ad7d-690671b7d6d8","Type":"ContainerDied","Data":"6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77"}
Sep 16 12:33:46.352778 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.352753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf" event={"ID":"e5cac034-1725-41a5-ad7d-690671b7d6d8","Type":"ContainerDied","Data":"033eb244c78e4d68b869668de876f83c718eeebb4933f2bc4edc12ab1e893c69"}
Sep 16 12:33:46.352778 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.352780    2560 scope.go:117] "RemoveContainer" containerID="6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77"
Sep 16 12:33:46.361583 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.361558    2560 scope.go:117] "RemoveContainer" containerID="6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77"
Sep 16 12:33:46.361877 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:33:46.361856    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77\": container with ID starting with 6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77 not found: ID does not exist" containerID="6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77"
Sep 16 12:33:46.361941 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.361888    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77"} err="failed to get container status \"6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77\": rpc error: code = NotFound desc = could not find container \"6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77\": container with ID starting with 6ac982a04e083d69a042bad457773eaee938191d70301d54617d481d271aae77 not found: ID does not exist"
Sep 16 12:33:46.408945 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.408923    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"]
Sep 16 12:33:46.416379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:46.416359    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-51ae50-56c4644758-jhzjf"]
Sep 16 12:33:48.324950 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:33:48.324917    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5cac034-1725-41a5-ad7d-690671b7d6d8" path="/var/lib/kubelet/pods/e5cac034-1725-41a5-ad7d-690671b7d6d8/volumes"
Sep 16 12:34:19.957845 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:19.957812    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"]
Sep 16 12:34:24.665556 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:24.665521    2560 scope.go:117] "RemoveContainer" containerID="d93ae451994d76878f73ed321e795c1dfd8353052d1e6c95fc8b4eaf0c1fc327"
Sep 16 12:34:33.048196 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.048165    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"]
Sep 16 12:34:33.048578 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.048564    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5cac034-1725-41a5-ad7d-690671b7d6d8" containerName="payload-pre-processing"
Sep 16 12:34:33.048622 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.048579    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5cac034-1725-41a5-ad7d-690671b7d6d8" containerName="payload-pre-processing"
Sep 16 12:34:33.048622 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.048600    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53874ddb-6613-4e6f-879d-1461c47d06e9" containerName="revoke-keys"
Sep 16 12:34:33.048622 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.048605    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="53874ddb-6613-4e6f-879d-1461c47d06e9" containerName="revoke-keys"
Sep 16 12:34:33.048756 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.048671    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5cac034-1725-41a5-ad7d-690671b7d6d8" containerName="payload-pre-processing"
Sep 16 12:34:33.048756 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.048679    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="53874ddb-6613-4e6f-879d-1461c47d06e9" containerName="revoke-keys"
Sep 16 12:34:33.051886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.051871    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.068632 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.068613    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a9008\"/\"sub-test-model-0a9008-kserve-self-signed-certs\""
Sep 16 12:34:33.137165 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.137139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"]
Sep 16 12:34:33.159280 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.159256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-model-cache\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.159399 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.159288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455cd13f-f838-4c78-8541-933ad87be44c-tls-certs\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.159399 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.159321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mrf\" (UniqueName: \"kubernetes.io/projected/455cd13f-f838-4c78-8541-933ad87be44c-kube-api-access-v8mrf\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.159495 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.159402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-home\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.159495 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.159453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-tmp-dir\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.159577 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.159505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-dshm\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260108 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mrf\" (UniqueName: \"kubernetes.io/projected/455cd13f-f838-4c78-8541-933ad87be44c-kube-api-access-v8mrf\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260218 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-home\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260218 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-tmp-dir\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260218 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-dshm\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260350 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-model-cache\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260350 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455cd13f-f838-4c78-8541-933ad87be44c-tls-certs\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260535 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-home\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260579 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-model-cache\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.260622 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.260594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-tmp-dir\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.262645 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.262617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-dshm\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.262748 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.262730    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455cd13f-f838-4c78-8541-933ad87be44c-tls-certs\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.270081 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.270063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mrf\" (UniqueName: \"kubernetes.io/projected/455cd13f-f838-4c78-8541-933ad87be44c-kube-api-access-v8mrf\") pod \"sub-test-model-0a9008-kserve-577b6454f9-tqmt6\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") " pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.361217 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.361194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:33.652432 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:33.652404    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"]
Sep 16 12:34:33.652432 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:34:33.652428    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod455cd13f_f838_4c78_8541_933ad87be44c.slice/crio-e41c170cfa58cb85a78a677d1b3cd74ce66a9425fa2819d75fb04375a86f7e34 WatchSource:0}: Error finding container e41c170cfa58cb85a78a677d1b3cd74ce66a9425fa2819d75fb04375a86f7e34: Status 404 returned error can't find the container with id e41c170cfa58cb85a78a677d1b3cd74ce66a9425fa2819d75fb04375a86f7e34
Sep 16 12:34:34.619026 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:34.618984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6" event={"ID":"455cd13f-f838-4c78-8541-933ad87be44c","Type":"ContainerStarted","Data":"31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb"}
Sep 16 12:34:34.619406 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:34.619044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6" event={"ID":"455cd13f-f838-4c78-8541-933ad87be44c","Type":"ContainerStarted","Data":"e41c170cfa58cb85a78a677d1b3cd74ce66a9425fa2819d75fb04375a86f7e34"}
Sep 16 12:34:34.642909 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:34.642863    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6" podStartSLOduration=1.642846986 podStartE2EDuration="1.642846986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:34.64261919 +0000 UTC m=+1870.777941173" watchObservedRunningTime="2026-09-16 12:34:34.642846986 +0000 UTC m=+1870.778169019"
Sep 16 12:34:34.977053 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:34.976960    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r" podUID="866d7f22-cadb-45f7-8d1d-01f3179877bd" containerName="main" containerID="cri-o://f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585" gracePeriod=45
Sep 16 12:34:35.227267 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.227204    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:34:35.379962 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.379927    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv4mn\" (UniqueName: \"kubernetes.io/projected/866d7f22-cadb-45f7-8d1d-01f3179877bd-kube-api-access-sv4mn\") pod \"866d7f22-cadb-45f7-8d1d-01f3179877bd\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") "
Sep 16 12:34:35.380110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.379986    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-model-cache\") pod \"866d7f22-cadb-45f7-8d1d-01f3179877bd\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") "
Sep 16 12:34:35.380110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380010    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866d7f22-cadb-45f7-8d1d-01f3179877bd-tls-certs\") pod \"866d7f22-cadb-45f7-8d1d-01f3179877bd\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") "
Sep 16 12:34:35.380110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380038    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-home\") pod \"866d7f22-cadb-45f7-8d1d-01f3179877bd\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") "
Sep 16 12:34:35.380110 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380107    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-tmp-dir\") pod \"866d7f22-cadb-45f7-8d1d-01f3179877bd\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") "
Sep 16 12:34:35.380343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380134    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-dshm\") pod \"866d7f22-cadb-45f7-8d1d-01f3179877bd\" (UID: \"866d7f22-cadb-45f7-8d1d-01f3179877bd\") "
Sep 16 12:34:35.380343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380278    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "866d7f22-cadb-45f7-8d1d-01f3179877bd" (UID: "866d7f22-cadb-45f7-8d1d-01f3179877bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:35.380439 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "866d7f22-cadb-45f7-8d1d-01f3179877bd" (UID: "866d7f22-cadb-45f7-8d1d-01f3179877bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:35.380439 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380375    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-home" (OuterVolumeSpecName: "home") pod "866d7f22-cadb-45f7-8d1d-01f3179877bd" (UID: "866d7f22-cadb-45f7-8d1d-01f3179877bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:35.380439 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.380421    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:34:35.382308 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.382280    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-dshm" (OuterVolumeSpecName: "dshm") pod "866d7f22-cadb-45f7-8d1d-01f3179877bd" (UID: "866d7f22-cadb-45f7-8d1d-01f3179877bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:35.382415 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.382329    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/866d7f22-cadb-45f7-8d1d-01f3179877bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "866d7f22-cadb-45f7-8d1d-01f3179877bd" (UID: "866d7f22-cadb-45f7-8d1d-01f3179877bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:34:35.382415 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.382351    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866d7f22-cadb-45f7-8d1d-01f3179877bd-kube-api-access-sv4mn" (OuterVolumeSpecName: "kube-api-access-sv4mn") pod "866d7f22-cadb-45f7-8d1d-01f3179877bd" (UID: "866d7f22-cadb-45f7-8d1d-01f3179877bd"). InnerVolumeSpecName "kube-api-access-sv4mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:35.481058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.481007    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866d7f22-cadb-45f7-8d1d-01f3179877bd-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:34:35.481058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.481025    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:34:35.481058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.481033    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:34:35.481058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.481041    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866d7f22-cadb-45f7-8d1d-01f3179877bd-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:34:35.481058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.481049    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4mn\" (UniqueName: \"kubernetes.io/projected/866d7f22-cadb-45f7-8d1d-01f3179877bd-kube-api-access-sv4mn\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:34:35.625988 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.625954    2560 generic.go:356] "Generic (PLEG): container finished" podID="866d7f22-cadb-45f7-8d1d-01f3179877bd" containerID="f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585" exitCode=0
Sep 16 12:34:35.626294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.625997    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"
Sep 16 12:34:35.626294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.626036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r" event={"ID":"866d7f22-cadb-45f7-8d1d-01f3179877bd","Type":"ContainerDied","Data":"f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585"}
Sep 16 12:34:35.626294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.626082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r" event={"ID":"866d7f22-cadb-45f7-8d1d-01f3179877bd","Type":"ContainerDied","Data":"0ecb39da163f23c9ce96add647cd703a29d764572b8748a9530fe4408efe7456"}
Sep 16 12:34:35.626294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.626099    2560 scope.go:117] "RemoveContainer" containerID="f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585"
Sep 16 12:34:35.635347 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.635333    2560 scope.go:117] "RemoveContainer" containerID="f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585"
Sep 16 12:34:35.635625 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:34:35.635605    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585\": container with ID starting with f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585 not found: ID does not exist" containerID="f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585"
Sep 16 12:34:35.635695 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.635636    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585"} err="failed to get container status \"f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585\": rpc error: code = NotFound desc = could not find container \"f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585\": container with ID starting with f1921b0712d758a9a321306baf4d1b3a71056f4d14c9e0bb4e61bd5252bfd585 not found: ID does not exist"
Sep 16 12:34:35.648374 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.648349    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"]
Sep 16 12:34:35.653127 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:35.653104    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/auth-test-model-0a9008-kserve-6695cd4747-nfl4r"]
Sep 16 12:34:36.326842 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:36.326805    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="866d7f22-cadb-45f7-8d1d-01f3179877bd" path="/var/lib/kubelet/pods/866d7f22-cadb-45f7-8d1d-01f3179877bd/volumes"
Sep 16 12:34:43.361961 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:43.361934    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:43.362336 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:43.361989    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:43.374022 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:43.373995    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:34:43.681087 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:34:43.681014    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:35:24.711185 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:24.705660    2560 scope.go:117] "RemoveContainer" containerID="05f0488ad9b0e4aacdcb7a813177477608763eab35e64b52b841af922e89269b"
Sep 16 12:35:43.202535 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:43.202502    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"]
Sep 16 12:35:58.221644 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.221599    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6" podUID="455cd13f-f838-4c78-8541-933ad87be44c" containerName="main" containerID="cri-o://31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb" gracePeriod=45
Sep 16 12:35:58.485436 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.485368    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:35:58.556586 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8mrf\" (UniqueName: \"kubernetes.io/projected/455cd13f-f838-4c78-8541-933ad87be44c-kube-api-access-v8mrf\") pod \"455cd13f-f838-4c78-8541-933ad87be44c\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") "
Sep 16 12:35:58.556745 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556597    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-dshm\") pod \"455cd13f-f838-4c78-8541-933ad87be44c\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") "
Sep 16 12:35:58.556745 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455cd13f-f838-4c78-8541-933ad87be44c-tls-certs\") pod \"455cd13f-f838-4c78-8541-933ad87be44c\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") "
Sep 16 12:35:58.556745 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556637    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-tmp-dir\") pod \"455cd13f-f838-4c78-8541-933ad87be44c\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") "
Sep 16 12:35:58.556745 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556672    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-home\") pod \"455cd13f-f838-4c78-8541-933ad87be44c\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") "
Sep 16 12:35:58.556745 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-model-cache\") pod \"455cd13f-f838-4c78-8541-933ad87be44c\" (UID: \"455cd13f-f838-4c78-8541-933ad87be44c\") "
Sep 16 12:35:58.557067 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "455cd13f-f838-4c78-8541-933ad87be44c" (UID: "455cd13f-f838-4c78-8541-933ad87be44c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:58.557067 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.556973    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-home" (OuterVolumeSpecName: "home") pod "455cd13f-f838-4c78-8541-933ad87be44c" (UID: "455cd13f-f838-4c78-8541-933ad87be44c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:58.557067 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.557014    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-model-cache" (OuterVolumeSpecName: "model-cache") pod "455cd13f-f838-4c78-8541-933ad87be44c" (UID: "455cd13f-f838-4c78-8541-933ad87be44c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:58.558932 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.558904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455cd13f-f838-4c78-8541-933ad87be44c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "455cd13f-f838-4c78-8541-933ad87be44c" (UID: "455cd13f-f838-4c78-8541-933ad87be44c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:35:58.559039 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.558932    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-dshm" (OuterVolumeSpecName: "dshm") pod "455cd13f-f838-4c78-8541-933ad87be44c" (UID: "455cd13f-f838-4c78-8541-933ad87be44c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:35:58.559039 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.558948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455cd13f-f838-4c78-8541-933ad87be44c-kube-api-access-v8mrf" (OuterVolumeSpecName: "kube-api-access-v8mrf") pod "455cd13f-f838-4c78-8541-933ad87be44c" (UID: "455cd13f-f838-4c78-8541-933ad87be44c"). InnerVolumeSpecName "kube-api-access-v8mrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:58.658074 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.658055    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mrf\" (UniqueName: \"kubernetes.io/projected/455cd13f-f838-4c78-8541-933ad87be44c-kube-api-access-v8mrf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:35:58.658074 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.658075    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:35:58.658216 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.658084    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455cd13f-f838-4c78-8541-933ad87be44c-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:35:58.658216 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.658094    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:35:58.658216 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.658101    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:35:58.658216 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:58.658109    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455cd13f-f838-4c78-8541-933ad87be44c-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:35:59.103106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.103070    2560 generic.go:356] "Generic (PLEG): container finished" podID="455cd13f-f838-4c78-8541-933ad87be44c" containerID="31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb" exitCode=0
Sep 16 12:35:59.103311 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.103131    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"
Sep 16 12:35:59.103311 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.103162    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6" event={"ID":"455cd13f-f838-4c78-8541-933ad87be44c","Type":"ContainerDied","Data":"31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb"}
Sep 16 12:35:59.103311 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.103209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6" event={"ID":"455cd13f-f838-4c78-8541-933ad87be44c","Type":"ContainerDied","Data":"e41c170cfa58cb85a78a677d1b3cd74ce66a9425fa2819d75fb04375a86f7e34"}
Sep 16 12:35:59.103311 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.103228    2560 scope.go:117] "RemoveContainer" containerID="31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb"
Sep 16 12:35:59.115467 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.115445    2560 scope.go:117] "RemoveContainer" containerID="31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb"
Sep 16 12:35:59.115765 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:35:59.115746    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb\": container with ID starting with 31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb not found: ID does not exist" containerID="31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb"
Sep 16 12:35:59.115854 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.115774    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb"} err="failed to get container status \"31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb\": rpc error: code = NotFound desc = could not find container \"31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb\": container with ID starting with 31fbd6383f75aeb931d6cbfd622b621b2f5e2e369e202c3623e3b4cdafac47cb not found: ID does not exist"
Sep 16 12:35:59.134494 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.134472    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"]
Sep 16 12:35:59.140045 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:35:59.140026    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a9008/sub-test-model-0a9008-kserve-577b6454f9-tqmt6"]
Sep 16 12:36:00.324365 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:00.324330    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455cd13f-f838-4c78-8541-933ad87be44c" path="/var/lib/kubelet/pods/455cd13f-f838-4c78-8541-933ad87be44c/volumes"
Sep 16 12:36:24.738693 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:24.738620    2560 scope.go:117] "RemoveContainer" containerID="a081b2031d905cb5e7d11545ae62240da24bd49bf82f0fe4a401291e1fc688cd"
Sep 16 12:36:24.747270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:24.747239    2560 scope.go:117] "RemoveContainer" containerID="3c64207c212dea2dccc4eaf0795f0058fa1deef5ad2e92f3cba7215ebd93ec03"
Sep 16 12:36:24.754883 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:24.754864    2560 scope.go:117] "RemoveContainer" containerID="a9647168c504f313e961852a94acce99804989ab6561e28eb1c0b280f1402ae9"
Sep 16 12:36:36.566074 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.566042    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"]
Sep 16 12:36:36.566436 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.566415    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455cd13f-f838-4c78-8541-933ad87be44c" containerName="main"
Sep 16 12:36:36.566436 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.566425    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="455cd13f-f838-4c78-8541-933ad87be44c" containerName="main"
Sep 16 12:36:36.566512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.566441    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="866d7f22-cadb-45f7-8d1d-01f3179877bd" containerName="main"
Sep 16 12:36:36.566512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.566447    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="866d7f22-cadb-45f7-8d1d-01f3179877bd" containerName="main"
Sep 16 12:36:36.566512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.566506    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="866d7f22-cadb-45f7-8d1d-01f3179877bd" containerName="main"
Sep 16 12:36:36.566618 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.566515    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="455cd13f-f838-4c78-8541-933ad87be44c" containerName="main"
Sep 16 12:36:36.569448 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.569431    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.574200 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.574159    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"kube-root-ca.crt\""
Sep 16 12:36:36.574349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.574229    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"openshift-service-ca.crt\""
Sep 16 12:36:36.574349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.574256    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"default-dockercfg-vbx8f\""
Sep 16 12:36:36.574349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.574229    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e94e5\"/\"rate-test-model-0e94e5-kserve-self-signed-certs\""
Sep 16 12:36:36.581307 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.581284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"]
Sep 16 12:36:36.595926 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.595905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-model-cache\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.596044 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.595967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5wf\" (UniqueName: \"kubernetes.io/projected/0ead60de-1f5b-4f86-b46f-e08a960c791a-kube-api-access-wl5wf\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.596044 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.596015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-home\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.596148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.596087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ead60de-1f5b-4f86-b46f-e08a960c791a-tls-certs\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.596148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.596127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-tmp-dir\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.596148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.596145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-dshm\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697147 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697122    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-home\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ead60de-1f5b-4f86-b46f-e08a960c791a-tls-certs\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-tmp-dir\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-dshm\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-model-cache\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697495 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5wf\" (UniqueName: \"kubernetes.io/projected/0ead60de-1f5b-4f86-b46f-e08a960c791a-kube-api-access-wl5wf\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-home\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697610 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-tmp-dir\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.697648 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.697615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-model-cache\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.699567 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.699548    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-dshm\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.699689 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.699670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ead60de-1f5b-4f86-b46f-e08a960c791a-tls-certs\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.705173 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.705156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5wf\" (UniqueName: \"kubernetes.io/projected/0ead60de-1f5b-4f86-b46f-e08a960c791a-kube-api-access-wl5wf\") pod \"rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") " pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:36.878586 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:36.878563    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:37.160224 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:37.160142    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"]
Sep 16 12:36:37.163200 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:36:37.163171    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ead60de_1f5b_4f86_b46f_e08a960c791a.slice/crio-37a9c591f02825bcd883c5e653d9da1d5c003d117b15af1da908c1c3274f5deb WatchSource:0}: Error finding container 37a9c591f02825bcd883c5e653d9da1d5c003d117b15af1da908c1c3274f5deb: Status 404 returned error can't find the container with id 37a9c591f02825bcd883c5e653d9da1d5c003d117b15af1da908c1c3274f5deb
Sep 16 12:36:37.314280 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:37.314251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv" event={"ID":"0ead60de-1f5b-4f86-b46f-e08a960c791a","Type":"ContainerStarted","Data":"7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906"}
Sep 16 12:36:37.314465 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:37.314288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv" event={"ID":"0ead60de-1f5b-4f86-b46f-e08a960c791a","Type":"ContainerStarted","Data":"37a9c591f02825bcd883c5e653d9da1d5c003d117b15af1da908c1c3274f5deb"}
Sep 16 12:36:37.333896 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:37.333851    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv" podStartSLOduration=1.333837742 podStartE2EDuration="1.333837742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:36:37.331757515 +0000 UTC m=+1993.467079481" watchObservedRunningTime="2026-09-16 12:36:37.333837742 +0000 UTC m=+1993.469159713"
Sep 16 12:36:46.879262 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:46.879224    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:46.879262 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:46.879269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:46.892004 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:46.891977    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:36:47.377558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:36:47.377531    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:37:12.258244 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:12.258208    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"]
Sep 16 12:37:19.713766 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.713720    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"]
Sep 16 12:37:19.718423 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.718397    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:19.721638 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.721618    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-869204-dockercfg-2vdcq\""
Sep 16 12:37:19.722296 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.722279    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-869204\""
Sep 16 12:37:19.729444 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.729421    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"]
Sep 16 12:37:19.859296 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.859267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a930391e-0efa-41c8-830e-67ea08ef65a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:19.859438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.859303    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9npdg\" (UniqueName: \"kubernetes.io/projected/a930391e-0efa-41c8-830e-67ea08ef65a2-kube-api-access-9npdg\") pod \"payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:19.960771 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.960741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a930391e-0efa-41c8-830e-67ea08ef65a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:19.960970 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.960815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9npdg\" (UniqueName: \"kubernetes.io/projected/a930391e-0efa-41c8-830e-67ea08ef65a2-kube-api-access-9npdg\") pod \"payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:19.961498 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.961479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a930391e-0efa-41c8-830e-67ea08ef65a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:19.969838 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:19.969762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9npdg\" (UniqueName: \"kubernetes.io/projected/a930391e-0efa-41c8-830e-67ea08ef65a2-kube-api-access-9npdg\") pod \"payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:20.032485 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:20.032464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:20.305208 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:20.305186    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"]
Sep 16 12:37:20.306695 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:37:20.306671    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda930391e_0efa_41c8_830e_67ea08ef65a2.slice/crio-66aa4c3ae930503e53f04b8a5d3d71d31222640e2f7c40780d2204a8d108c83e WatchSource:0}: Error finding container 66aa4c3ae930503e53f04b8a5d3d71d31222640e2f7c40780d2204a8d108c83e: Status 404 returned error can't find the container with id 66aa4c3ae930503e53f04b8a5d3d71d31222640e2f7c40780d2204a8d108c83e
Sep 16 12:37:20.540081 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:20.540046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7" event={"ID":"a930391e-0efa-41c8-830e-67ea08ef65a2","Type":"ContainerStarted","Data":"ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d"}
Sep 16 12:37:20.540264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:20.540106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7" event={"ID":"a930391e-0efa-41c8-830e-67ea08ef65a2","Type":"ContainerStarted","Data":"66aa4c3ae930503e53f04b8a5d3d71d31222640e2f7c40780d2204a8d108c83e"}
Sep 16 12:37:20.540264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:20.540182    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:20.561034 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:20.560950    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7" podStartSLOduration=1.560935269 podStartE2EDuration="1.560935269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:20.558338237 +0000 UTC m=+2036.693660208" watchObservedRunningTime="2026-09-16 12:37:20.560935269 +0000 UTC m=+2036.696257236"
Sep 16 12:37:27.278266 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.278201    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv" podUID="0ead60de-1f5b-4f86-b46f-e08a960c791a" containerName="main" containerID="cri-o://7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906" gracePeriod=45
Sep 16 12:37:27.366633 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.366600    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv" podUID="0ead60de-1f5b-4f86-b46f-e08a960c791a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 12:37:27.530307 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.530249    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:37:27.578569 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.578541    2560 generic.go:356] "Generic (PLEG): container finished" podID="0ead60de-1f5b-4f86-b46f-e08a960c791a" containerID="7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906" exitCode=0
Sep 16 12:37:27.578724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.578581    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"
Sep 16 12:37:27.578724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.578607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv" event={"ID":"0ead60de-1f5b-4f86-b46f-e08a960c791a","Type":"ContainerDied","Data":"7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906"}
Sep 16 12:37:27.578724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.578645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv" event={"ID":"0ead60de-1f5b-4f86-b46f-e08a960c791a","Type":"ContainerDied","Data":"37a9c591f02825bcd883c5e653d9da1d5c003d117b15af1da908c1c3274f5deb"}
Sep 16 12:37:27.578724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.578669    2560 scope.go:117] "RemoveContainer" containerID="7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906"
Sep 16 12:37:27.589358 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.589340    2560 scope.go:117] "RemoveContainer" containerID="7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906"
Sep 16 12:37:27.589664 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:37:27.589631    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906\": container with ID starting with 7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906 not found: ID does not exist" containerID="7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906"
Sep 16 12:37:27.589757 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.589668    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906"} err="failed to get container status \"7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906\": rpc error: code = NotFound desc = could not find container \"7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906\": container with ID starting with 7391e93811f25b69ccb9756e90ff912ab3c732a6c34ab071d5d38ef545a0a906 not found: ID does not exist"
Sep 16 12:37:27.631971 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.631946    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-home\") pod \"0ead60de-1f5b-4f86-b46f-e08a960c791a\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") "
Sep 16 12:37:27.632071 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.631984    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-tmp-dir\") pod \"0ead60de-1f5b-4f86-b46f-e08a960c791a\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") "
Sep 16 12:37:27.632071 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632022    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ead60de-1f5b-4f86-b46f-e08a960c791a-tls-certs\") pod \"0ead60de-1f5b-4f86-b46f-e08a960c791a\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") "
Sep 16 12:37:27.632071 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632042    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl5wf\" (UniqueName: \"kubernetes.io/projected/0ead60de-1f5b-4f86-b46f-e08a960c791a-kube-api-access-wl5wf\") pod \"0ead60de-1f5b-4f86-b46f-e08a960c791a\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") "
Sep 16 12:37:27.632230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-dshm\") pod \"0ead60de-1f5b-4f86-b46f-e08a960c791a\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") "
Sep 16 12:37:27.632230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632150    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-model-cache\") pod \"0ead60de-1f5b-4f86-b46f-e08a960c791a\" (UID: \"0ead60de-1f5b-4f86-b46f-e08a960c791a\") "
Sep 16 12:37:27.632230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632206    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-home" (OuterVolumeSpecName: "home") pod "0ead60de-1f5b-4f86-b46f-e08a960c791a" (UID: "0ead60de-1f5b-4f86-b46f-e08a960c791a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:27.632393 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632297    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ead60de-1f5b-4f86-b46f-e08a960c791a" (UID: "0ead60de-1f5b-4f86-b46f-e08a960c791a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:27.632496 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632474    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ead60de-1f5b-4f86-b46f-e08a960c791a" (UID: "0ead60de-1f5b-4f86-b46f-e08a960c791a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:27.632538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632495    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:37:27.632538 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.632511    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:37:27.634346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.634315    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ead60de-1f5b-4f86-b46f-e08a960c791a-kube-api-access-wl5wf" (OuterVolumeSpecName: "kube-api-access-wl5wf") pod "0ead60de-1f5b-4f86-b46f-e08a960c791a" (UID: "0ead60de-1f5b-4f86-b46f-e08a960c791a"). InnerVolumeSpecName "kube-api-access-wl5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:37:27.634455 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.634370    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ead60de-1f5b-4f86-b46f-e08a960c791a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ead60de-1f5b-4f86-b46f-e08a960c791a" (UID: "0ead60de-1f5b-4f86-b46f-e08a960c791a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:37:27.634455 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.634400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-dshm" (OuterVolumeSpecName: "dshm") pod "0ead60de-1f5b-4f86-b46f-e08a960c791a" (UID: "0ead60de-1f5b-4f86-b46f-e08a960c791a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:27.733384 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.733362    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ead60de-1f5b-4f86-b46f-e08a960c791a-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:37:27.733384 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.733385    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5wf\" (UniqueName: \"kubernetes.io/projected/0ead60de-1f5b-4f86-b46f-e08a960c791a-kube-api-access-wl5wf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:37:27.733529 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.733396    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:37:27.733529 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.733405    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ead60de-1f5b-4f86-b46f-e08a960c791a-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:37:27.902893 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.902867    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"]
Sep 16 12:37:27.909985 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:27.909959    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e94e5/rate-test-model-0e94e5-kserve-bdf8cb6d7-ln4cv"]
Sep 16 12:37:28.325006 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:28.324934    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ead60de-1f5b-4f86-b46f-e08a960c791a" path="/var/lib/kubelet/pods/0ead60de-1f5b-4f86-b46f-e08a960c791a/volumes"
Sep 16 12:37:31.548600 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:31.548569    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:37:49.740286 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.740199    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"]
Sep 16 12:37:49.740886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.740621    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ead60de-1f5b-4f86-b46f-e08a960c791a" containerName="main"
Sep 16 12:37:49.740886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.740635    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ead60de-1f5b-4f86-b46f-e08a960c791a" containerName="main"
Sep 16 12:37:49.740886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.740710    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ead60de-1f5b-4f86-b46f-e08a960c791a" containerName="main"
Sep 16 12:37:49.744065 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.744046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:49.747247 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.747225    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-89b7ca-dockercfg-khhnn\""
Sep 16 12:37:49.747379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.747225    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-89b7ca\""
Sep 16 12:37:49.752573 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.752552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"]
Sep 16 12:37:49.807466 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.807437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmnq\" (UniqueName: \"kubernetes.io/projected/aa718987-1722-4f84-af4c-d048b7eedda8-kube-api-access-rbmnq\") pod \"payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:49.807594 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.807481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa718987-1722-4f84-af4c-d048b7eedda8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:49.827306 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.827279    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"]
Sep 16 12:37:49.830769 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.830753    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:49.842814 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.842762    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"]
Sep 16 12:37:49.908188 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.908159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e75adf-f968-4d5c-adc1-b9d78ab90520-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:49.908306 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.908208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmnq\" (UniqueName: \"kubernetes.io/projected/aa718987-1722-4f84-af4c-d048b7eedda8-kube-api-access-rbmnq\") pod \"payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:49.908306 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.908266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa718987-1722-4f84-af4c-d048b7eedda8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:49.908385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.908345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724d4\" (UniqueName: \"kubernetes.io/projected/f6e75adf-f968-4d5c-adc1-b9d78ab90520-kube-api-access-724d4\") pod \"payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:49.908854 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.908836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa718987-1722-4f84-af4c-d048b7eedda8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:49.916532 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:49.916512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmnq\" (UniqueName: \"kubernetes.io/projected/aa718987-1722-4f84-af4c-d048b7eedda8-kube-api-access-rbmnq\") pod \"payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:50.009724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.009652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724d4\" (UniqueName: \"kubernetes.io/projected/f6e75adf-f968-4d5c-adc1-b9d78ab90520-kube-api-access-724d4\") pod \"payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:50.009724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.009718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e75adf-f968-4d5c-adc1-b9d78ab90520-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:50.010379 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.010362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e75adf-f968-4d5c-adc1-b9d78ab90520-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:50.018741 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.018718    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-724d4\" (UniqueName: \"kubernetes.io/projected/f6e75adf-f968-4d5c-adc1-b9d78ab90520-kube-api-access-724d4\") pod \"payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:50.055507 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.055480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:50.142162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.142137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:50.327333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.327312    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"]
Sep 16 12:37:50.328987 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:37:50.328962    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa718987_1722_4f84_af4c_d048b7eedda8.slice/crio-e2c3f88d050ff17001c36303d0ef3bc4fe9008edcdc20c523b2ce3d77216f32d WatchSource:0}: Error finding container e2c3f88d050ff17001c36303d0ef3bc4fe9008edcdc20c523b2ce3d77216f32d: Status 404 returned error can't find the container with id e2c3f88d050ff17001c36303d0ef3bc4fe9008edcdc20c523b2ce3d77216f32d
Sep 16 12:37:50.416975 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.413324    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"]
Sep 16 12:37:50.417744 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:37:50.417714    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e75adf_f968_4d5c_adc1_b9d78ab90520.slice/crio-f6c019f333949f4fe414faf03a8a3e1f3ed30f6be4b86061e436571bb33045c9 WatchSource:0}: Error finding container f6c019f333949f4fe414faf03a8a3e1f3ed30f6be4b86061e436571bb33045c9: Status 404 returned error can't find the container with id f6c019f333949f4fe414faf03a8a3e1f3ed30f6be4b86061e436571bb33045c9
Sep 16 12:37:50.707953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.707904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df" event={"ID":"aa718987-1722-4f84-af4c-d048b7eedda8","Type":"ContainerStarted","Data":"73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750"}
Sep 16 12:37:50.708148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.708010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df" event={"ID":"aa718987-1722-4f84-af4c-d048b7eedda8","Type":"ContainerStarted","Data":"e2c3f88d050ff17001c36303d0ef3bc4fe9008edcdc20c523b2ce3d77216f32d"}
Sep 16 12:37:50.708148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.708041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:37:50.709851 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.709823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2" event={"ID":"f6e75adf-f968-4d5c-adc1-b9d78ab90520","Type":"ContainerStarted","Data":"7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a"}
Sep 16 12:37:50.709992 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.709875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2" event={"ID":"f6e75adf-f968-4d5c-adc1-b9d78ab90520","Type":"ContainerStarted","Data":"f6c019f333949f4fe414faf03a8a3e1f3ed30f6be4b86061e436571bb33045c9"}
Sep 16 12:37:50.709992 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.709905    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:37:50.724346 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.724303    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df" podStartSLOduration=1.7242871960000001 podStartE2EDuration="1.724287196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:50.72418081 +0000 UTC m=+2066.859502783" watchObservedRunningTime="2026-09-16 12:37:50.724287196 +0000 UTC m=+2066.859609171"
Sep 16 12:37:50.742936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:37:50.742897    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2" podStartSLOduration=1.742879385 podStartE2EDuration="1.742879385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:50.741849242 +0000 UTC m=+2066.877171206" watchObservedRunningTime="2026-09-16 12:37:50.742879385 +0000 UTC m=+2066.878201355"
Sep 16 12:38:01.717452 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:01.717421    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:38:01.717848 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:01.717491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:38:20.926856 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:20.926816    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8hfz8_8120361a-3d74-47be-b9a8-5ece86158766/payload-processing/0.log"
Sep 16 12:38:21.042994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:21.042964    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2_f6e75adf-f968-4d5c-adc1-b9d78ab90520/payload-processing/0.log"
Sep 16 12:38:24.462264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:24.462232    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:38:24.468482 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:24.468458    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:38:24.795697 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:24.795611    2560 scope.go:117] "RemoveContainer" containerID="54bef9dc565d94818e27cb20663574829962fbee6c6f1a814e9847a8679cd61e"
Sep 16 12:38:24.804641 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:24.804621    2560 scope.go:117] "RemoveContainer" containerID="904701144c10549f840853ccedbc8b20ceec1217a8d1b73a2a02c601613349fb"
Sep 16 12:38:28.716821 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.716766    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"]
Sep 16 12:38:28.720530 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.720509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.723735 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.723716    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-869204\"/\"ipp-route-869204-kserve-self-signed-certs\""
Sep 16 12:38:28.723735 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.723728    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-869204\"/\"kube-root-ca.crt\""
Sep 16 12:38:28.724980 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.724962    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-869204\"/\"default-dockercfg-22k5b\""
Sep 16 12:38:28.725099 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.724968    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-869204\"/\"openshift-service-ca.crt\""
Sep 16 12:38:28.730123 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.730102    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"]
Sep 16 12:38:28.756442 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.756423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-home\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.756559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.756451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p6vd\" (UniqueName: \"kubernetes.io/projected/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-kube-api-access-9p6vd\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.756559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.756505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tmp-dir\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.756640 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.756605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-dshm\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.756640 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.756632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-model-cache\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.756718 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.756650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tls-certs\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.857711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.857683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-dshm\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.857890 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.857718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-model-cache\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.857890 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.857736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tls-certs\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.857890 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.857751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-home\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.857890 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.857768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p6vd\" (UniqueName: \"kubernetes.io/projected/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-kube-api-access-9p6vd\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.857890 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.857856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tmp-dir\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.858198 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.858176    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-home\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.858198 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.858188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-model-cache\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.858303 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.858228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tmp-dir\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.860315 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.860295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-dshm\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.860484 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.860467    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tls-certs\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:28.866854 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:28.866835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p6vd\" (UniqueName: \"kubernetes.io/projected/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-kube-api-access-9p6vd\") pod \"ipp-route-869204-kserve-77486654f5-lgxlz\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") " pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:29.031113 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:29.031038    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:29.308366 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:29.308090    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"]
Sep 16 12:38:29.922353 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:29.922319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz" event={"ID":"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4","Type":"ContainerStarted","Data":"fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921"}
Sep 16 12:38:29.922721 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:29.922359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz" event={"ID":"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4","Type":"ContainerStarted","Data":"9a1bdc9e94465e3136f5e8f2cb745d5ed1ae108f51ece22bd642e826957542a0"}
Sep 16 12:38:29.943084 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:29.943032    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz" podStartSLOduration=1.9430184910000001 podStartE2EDuration="1.943018491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:29.940499572 +0000 UTC m=+2106.075821555" watchObservedRunningTime="2026-09-16 12:38:29.943018491 +0000 UTC m=+2106.078340461"
Sep 16 12:38:39.031820 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:39.031770    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:39.032302 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:39.031828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:39.044509 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:39.044486    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:39.986356 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:39.986329    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:38:57.022981 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:57.022946    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8hfz8_8120361a-3d74-47be-b9a8-5ece86158766/payload-processing/0.log"
Sep 16 12:38:57.148714 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:38:57.148687    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2_f6e75adf-f968-4d5c-adc1-b9d78ab90520/payload-processing/0.log"
Sep 16 12:39:05.563705 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.563670    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"]
Sep 16 12:39:05.567464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.567444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:05.570459 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.570441    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b03a3f-dockercfg-92ncs\""
Sep 16 12:39:05.570563 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.570441    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b03a3f\""
Sep 16 12:39:05.582229 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.582205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"]
Sep 16 12:39:05.668181 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.668158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqs9\" (UniqueName: \"kubernetes.io/projected/e94863bb-20e8-4106-9d73-5aae2816398b-kube-api-access-4nqs9\") pod \"payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:05.668294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.668239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94863bb-20e8-4106-9d73-5aae2816398b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:05.769264 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.769239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94863bb-20e8-4106-9d73-5aae2816398b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:05.769388 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.769290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqs9\" (UniqueName: \"kubernetes.io/projected/e94863bb-20e8-4106-9d73-5aae2816398b-kube-api-access-4nqs9\") pod \"payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:05.770005 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.769982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94863bb-20e8-4106-9d73-5aae2816398b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:05.779438 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.779419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqs9\" (UniqueName: \"kubernetes.io/projected/e94863bb-20e8-4106-9d73-5aae2816398b-kube-api-access-4nqs9\") pod \"payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:05.878165 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:05.878140    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:06.157773 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:06.157713    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"]
Sep 16 12:39:06.160249 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:39:06.160219    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94863bb_20e8_4106_9d73_5aae2816398b.slice/crio-155a8ef7aca37e34e798a6cbd6dc280663bcb68fe0746ae2688dbbfe8ddeb22d WatchSource:0}: Error finding container 155a8ef7aca37e34e798a6cbd6dc280663bcb68fe0746ae2688dbbfe8ddeb22d: Status 404 returned error can't find the container with id 155a8ef7aca37e34e798a6cbd6dc280663bcb68fe0746ae2688dbbfe8ddeb22d
Sep 16 12:39:07.128673 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:07.128640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt" event={"ID":"e94863bb-20e8-4106-9d73-5aae2816398b","Type":"ContainerStarted","Data":"b836c0043eddc79be7f730dc71d98ce0798ab5cfb3662c1b105e91a9c60f98cc"}
Sep 16 12:39:07.129082 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:07.128699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt" event={"ID":"e94863bb-20e8-4106-9d73-5aae2816398b","Type":"ContainerStarted","Data":"155a8ef7aca37e34e798a6cbd6dc280663bcb68fe0746ae2688dbbfe8ddeb22d"}
Sep 16 12:39:07.129082 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:07.128725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:07.148551 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:07.148501    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt" podStartSLOduration=2.148486154 podStartE2EDuration="2.148486154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:07.148050591 +0000 UTC m=+2143.283372579" watchObservedRunningTime="2026-09-16 12:39:07.148486154 +0000 UTC m=+2143.283808125"
Sep 16 12:39:18.137234 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:18.137153    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:24.836191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:24.836158    2560 scope.go:117] "RemoveContainer" containerID="0e9dd481894fc1ea21a0ca3be937eadace12a140a626d43145687b29e2a95b94"
Sep 16 12:39:24.845186 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:24.845164    2560 scope.go:117] "RemoveContainer" containerID="ad7bd095317c942bf7bb5378a02bb9ee7a2200d950491b6cce001fc2c37b7375"
Sep 16 12:39:24.852891 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:24.852874    2560 scope.go:117] "RemoveContainer" containerID="c1b681d861df3907b76b5fae46abb9aaa42a9da10848d1de413be0b2bf39b9ae"
Sep 16 12:39:31.981253 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:31.981217    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"]
Sep 16 12:39:31.985054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:31.985026    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"
Sep 16 12:39:31.987673 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:31.987652    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:39:31.987838 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:31.987816    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:39:31.992473 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:31.992445    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"]
Sep 16 12:39:31.997426 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:31.997403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4237696-27a9-4441-8f85-ece77e631bb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98\" (UID: \"c4237696-27a9-4441-8f85-ece77e631bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"
Sep 16 12:39:32.098742 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:32.098712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4237696-27a9-4441-8f85-ece77e631bb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98\" (UID: \"c4237696-27a9-4441-8f85-ece77e631bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"
Sep 16 12:39:32.099380 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:32.099362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4237696-27a9-4441-8f85-ece77e631bb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98\" (UID: \"c4237696-27a9-4441-8f85-ece77e631bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"
Sep 16 12:39:32.298063 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:32.297999    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"
Sep 16 12:39:32.575614 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:32.575591    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"]
Sep 16 12:39:32.577318 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:39:32.577279    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4237696_27a9_4441_8f85_ece77e631bb9.slice/crio-e0f0fa2e67c8c37f620c5ac29baa3b11aff0ecf8791291a87827d9acdf2638bb WatchSource:0}: Error finding container e0f0fa2e67c8c37f620c5ac29baa3b11aff0ecf8791291a87827d9acdf2638bb: Status 404 returned error can't find the container with id e0f0fa2e67c8c37f620c5ac29baa3b11aff0ecf8791291a87827d9acdf2638bb
Sep 16 12:39:33.272416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:33.272330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98" event={"ID":"c4237696-27a9-4441-8f85-ece77e631bb9","Type":"ContainerStarted","Data":"fd6c007cb79121cfd04bc86e6e15a04717d97b141732f87a018ef48636c66154"}
Sep 16 12:39:33.272416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:33.272388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98" event={"ID":"c4237696-27a9-4441-8f85-ece77e631bb9","Type":"ContainerStarted","Data":"e0f0fa2e67c8c37f620c5ac29baa3b11aff0ecf8791291a87827d9acdf2638bb"}
Sep 16 12:39:33.290705 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:33.290658    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98" podStartSLOduration=2.290642594 podStartE2EDuration="2.290642594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:33.28803145 +0000 UTC m=+2169.423353422" watchObservedRunningTime="2026-09-16 12:39:33.290642594 +0000 UTC m=+2169.425964562"
Sep 16 12:39:34.279269 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:34.279230    2560 generic.go:356] "Generic (PLEG): container finished" podID="c4237696-27a9-4441-8f85-ece77e631bb9" containerID="fd6c007cb79121cfd04bc86e6e15a04717d97b141732f87a018ef48636c66154" exitCode=0
Sep 16 12:39:34.279915 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:34.279324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98" event={"ID":"c4237696-27a9-4441-8f85-ece77e631bb9","Type":"ContainerDied","Data":"fd6c007cb79121cfd04bc86e6e15a04717d97b141732f87a018ef48636c66154"}
Sep 16 12:39:35.423008 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:35.422981    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"
Sep 16 12:39:35.528083 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:35.528057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4237696-27a9-4441-8f85-ece77e631bb9-maas-api-service-ca\") pod \"c4237696-27a9-4441-8f85-ece77e631bb9\" (UID: \"c4237696-27a9-4441-8f85-ece77e631bb9\") "
Sep 16 12:39:35.528362 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:35.528341    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4237696-27a9-4441-8f85-ece77e631bb9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4237696-27a9-4441-8f85-ece77e631bb9" (UID: "c4237696-27a9-4441-8f85-ece77e631bb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:35.629446 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:35.629421    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4237696-27a9-4441-8f85-ece77e631bb9-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:39:36.291500 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:36.291464    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"
Sep 16 12:39:36.291677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:36.291509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98" event={"ID":"c4237696-27a9-4441-8f85-ece77e631bb9","Type":"ContainerDied","Data":"e0f0fa2e67c8c37f620c5ac29baa3b11aff0ecf8791291a87827d9acdf2638bb"}
Sep 16 12:39:36.291677 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:36.291560    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f0fa2e67c8c37f620c5ac29baa3b11aff0ecf8791291a87827d9acdf2638bb"
Sep 16 12:39:43.127868 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.127828    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"]
Sep 16 12:39:43.128345 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.128084    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt" podUID="e94863bb-20e8-4106-9d73-5aae2816398b" containerName="payload-processing" containerID="cri-o://b836c0043eddc79be7f730dc71d98ce0798ab5cfb3662c1b105e91a9c60f98cc" gracePeriod=30
Sep 16 12:39:43.334818 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.334769    2560 generic.go:356] "Generic (PLEG): container finished" podID="e94863bb-20e8-4106-9d73-5aae2816398b" containerID="b836c0043eddc79be7f730dc71d98ce0798ab5cfb3662c1b105e91a9c60f98cc" exitCode=0
Sep 16 12:39:43.334959 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.334819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt" event={"ID":"e94863bb-20e8-4106-9d73-5aae2816398b","Type":"ContainerDied","Data":"b836c0043eddc79be7f730dc71d98ce0798ab5cfb3662c1b105e91a9c60f98cc"}
Sep 16 12:39:43.341756 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.341726    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"]
Sep 16 12:39:43.348651 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.348629    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b03a3f-qwg98"]
Sep 16 12:39:43.385873 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.385819    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:43.493854 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.493826    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nqs9\" (UniqueName: \"kubernetes.io/projected/e94863bb-20e8-4106-9d73-5aae2816398b-kube-api-access-4nqs9\") pod \"e94863bb-20e8-4106-9d73-5aae2816398b\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") "
Sep 16 12:39:43.493990 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.493896    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94863bb-20e8-4106-9d73-5aae2816398b-plugins-config-volume\") pod \"e94863bb-20e8-4106-9d73-5aae2816398b\" (UID: \"e94863bb-20e8-4106-9d73-5aae2816398b\") "
Sep 16 12:39:43.494229 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.494209    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94863bb-20e8-4106-9d73-5aae2816398b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e94863bb-20e8-4106-9d73-5aae2816398b" (UID: "e94863bb-20e8-4106-9d73-5aae2816398b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:43.496103 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.496078    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94863bb-20e8-4106-9d73-5aae2816398b-kube-api-access-4nqs9" (OuterVolumeSpecName: "kube-api-access-4nqs9") pod "e94863bb-20e8-4106-9d73-5aae2816398b" (UID: "e94863bb-20e8-4106-9d73-5aae2816398b"). InnerVolumeSpecName "kube-api-access-4nqs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:43.595558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.595533    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94863bb-20e8-4106-9d73-5aae2816398b-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:39:43.595558 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:43.595558    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqs9\" (UniqueName: \"kubernetes.io/projected/e94863bb-20e8-4106-9d73-5aae2816398b-kube-api-access-4nqs9\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:39:44.324739 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:44.324707    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4237696-27a9-4441-8f85-ece77e631bb9" path="/var/lib/kubelet/pods/c4237696-27a9-4441-8f85-ece77e631bb9/volumes"
Sep 16 12:39:44.342315 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:44.342285    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"
Sep 16 12:39:44.342461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:44.342330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt" event={"ID":"e94863bb-20e8-4106-9d73-5aae2816398b","Type":"ContainerDied","Data":"155a8ef7aca37e34e798a6cbd6dc280663bcb68fe0746ae2688dbbfe8ddeb22d"}
Sep 16 12:39:44.342461 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:44.342399    2560 scope.go:117] "RemoveContainer" containerID="b836c0043eddc79be7f730dc71d98ce0798ab5cfb3662c1b105e91a9c60f98cc"
Sep 16 12:39:44.364413 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:44.364384    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"]
Sep 16 12:39:44.366230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:44.366210    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b03a3f-5dd7ddd4bf-42tgt"]
Sep 16 12:39:46.325589 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:46.325555    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94863bb-20e8-4106-9d73-5aae2816398b" path="/var/lib/kubelet/pods/e94863bb-20e8-4106-9d73-5aae2816398b/volumes"
Sep 16 12:39:53.861334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.861299    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"]
Sep 16 12:39:53.861728 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.861716    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4237696-27a9-4441-8f85-ece77e631bb9" containerName="revoke-keys"
Sep 16 12:39:53.861772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.861730    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4237696-27a9-4441-8f85-ece77e631bb9" containerName="revoke-keys"
Sep 16 12:39:53.861772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.861743    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94863bb-20e8-4106-9d73-5aae2816398b" containerName="payload-processing"
Sep 16 12:39:53.861772 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.861748    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94863bb-20e8-4106-9d73-5aae2816398b" containerName="payload-processing"
Sep 16 12:39:53.861893 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.861819    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4237696-27a9-4441-8f85-ece77e631bb9" containerName="revoke-keys"
Sep 16 12:39:53.861893 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.861829    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94863bb-20e8-4106-9d73-5aae2816398b" containerName="payload-processing"
Sep 16 12:39:53.866436 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.866419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"
Sep 16 12:39:53.869659 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.869636    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:39:53.869803 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.869673    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:39:53.873381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.873358    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"]
Sep 16 12:39:53.984297 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:53.984268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a98157-405a-41cf-877b-ef4f13ed00ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r\" (UID: \"e9a98157-405a-41cf-877b-ef4f13ed00ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"
Sep 16 12:39:54.084886 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:54.084861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a98157-405a-41cf-877b-ef4f13ed00ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r\" (UID: \"e9a98157-405a-41cf-877b-ef4f13ed00ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"
Sep 16 12:39:54.085429 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:54.085412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a98157-405a-41cf-877b-ef4f13ed00ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r\" (UID: \"e9a98157-405a-41cf-877b-ef4f13ed00ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"
Sep 16 12:39:54.179556 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:54.179495    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"
Sep 16 12:39:54.455364 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:54.455340    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"]
Sep 16 12:39:54.457356 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:39:54.457328    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a98157_405a_41cf_877b_ef4f13ed00ce.slice/crio-24aaca14b4aa457b50dfeacf76928af51e5383c22944eed26cdf18e3ee967771 WatchSource:0}: Error finding container 24aaca14b4aa457b50dfeacf76928af51e5383c22944eed26cdf18e3ee967771: Status 404 returned error can't find the container with id 24aaca14b4aa457b50dfeacf76928af51e5383c22944eed26cdf18e3ee967771
Sep 16 12:39:55.400335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:55.400244    2560 generic.go:356] "Generic (PLEG): container finished" podID="e9a98157-405a-41cf-877b-ef4f13ed00ce" containerID="2d78778428407d3a2ca0b1aac793077b6808398f433f8b74623033168baeffea" exitCode=0
Sep 16 12:39:55.400728 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:55.400335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r" event={"ID":"e9a98157-405a-41cf-877b-ef4f13ed00ce","Type":"ContainerDied","Data":"2d78778428407d3a2ca0b1aac793077b6808398f433f8b74623033168baeffea"}
Sep 16 12:39:55.400728 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:55.400392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r" event={"ID":"e9a98157-405a-41cf-877b-ef4f13ed00ce","Type":"ContainerStarted","Data":"24aaca14b4aa457b50dfeacf76928af51e5383c22944eed26cdf18e3ee967771"}
Sep 16 12:39:56.537055 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:56.537031    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"
Sep 16 12:39:56.606310 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:56.606286    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a98157-405a-41cf-877b-ef4f13ed00ce-maas-api-service-ca\") pod \"e9a98157-405a-41cf-877b-ef4f13ed00ce\" (UID: \"e9a98157-405a-41cf-877b-ef4f13ed00ce\") "
Sep 16 12:39:56.606614 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:56.606592    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a98157-405a-41cf-877b-ef4f13ed00ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9a98157-405a-41cf-877b-ef4f13ed00ce" (UID: "e9a98157-405a-41cf-877b-ef4f13ed00ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:56.707070 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:56.707019    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a98157-405a-41cf-877b-ef4f13ed00ce-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:39:57.412408 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:57.412378    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"
Sep 16 12:39:57.412582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:57.412378    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r" event={"ID":"e9a98157-405a-41cf-877b-ef4f13ed00ce","Type":"ContainerDied","Data":"24aaca14b4aa457b50dfeacf76928af51e5383c22944eed26cdf18e3ee967771"}
Sep 16 12:39:57.412582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:39:57.412504    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24aaca14b4aa457b50dfeacf76928af51e5383c22944eed26cdf18e3ee967771"
Sep 16 12:40:09.641681 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:09.641645    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"]
Sep 16 12:40:09.642287 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:09.641900    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7" podUID="a930391e-0efa-41c8-830e-67ea08ef65a2" containerName="payload-pre-processing" containerID="cri-o://ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d" gracePeriod=30
Sep 16 12:40:09.798412 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:09.798378    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"]
Sep 16 12:40:09.802971 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:09.802942    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-869204-2lk6r"]
Sep 16 12:40:09.906343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:09.906285    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:40:10.021315 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.021285    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a930391e-0efa-41c8-830e-67ea08ef65a2-plugins-config-volume\") pod \"a930391e-0efa-41c8-830e-67ea08ef65a2\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") "
Sep 16 12:40:10.021486 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.021350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9npdg\" (UniqueName: \"kubernetes.io/projected/a930391e-0efa-41c8-830e-67ea08ef65a2-kube-api-access-9npdg\") pod \"a930391e-0efa-41c8-830e-67ea08ef65a2\" (UID: \"a930391e-0efa-41c8-830e-67ea08ef65a2\") "
Sep 16 12:40:10.021700 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.021667    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a930391e-0efa-41c8-830e-67ea08ef65a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a930391e-0efa-41c8-830e-67ea08ef65a2" (UID: "a930391e-0efa-41c8-830e-67ea08ef65a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:40:10.023631 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.023607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a930391e-0efa-41c8-830e-67ea08ef65a2-kube-api-access-9npdg" (OuterVolumeSpecName: "kube-api-access-9npdg") pod "a930391e-0efa-41c8-830e-67ea08ef65a2" (UID: "a930391e-0efa-41c8-830e-67ea08ef65a2"). InnerVolumeSpecName "kube-api-access-9npdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:40:10.122466 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.122443    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a930391e-0efa-41c8-830e-67ea08ef65a2-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.122466 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.122465    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9npdg\" (UniqueName: \"kubernetes.io/projected/a930391e-0efa-41c8-830e-67ea08ef65a2-kube-api-access-9npdg\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:10.324582 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.324499    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a98157-405a-41cf-877b-ef4f13ed00ce" path="/var/lib/kubelet/pods/e9a98157-405a-41cf-877b-ef4f13ed00ce/volumes"
Sep 16 12:40:10.485680 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.485644    2560 generic.go:356] "Generic (PLEG): container finished" podID="a930391e-0efa-41c8-830e-67ea08ef65a2" containerID="ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d" exitCode=0
Sep 16 12:40:10.485908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.485707    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"
Sep 16 12:40:10.485908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.485721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7" event={"ID":"a930391e-0efa-41c8-830e-67ea08ef65a2","Type":"ContainerDied","Data":"ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d"}
Sep 16 12:40:10.485908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.485782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7" event={"ID":"a930391e-0efa-41c8-830e-67ea08ef65a2","Type":"ContainerDied","Data":"66aa4c3ae930503e53f04b8a5d3d71d31222640e2f7c40780d2204a8d108c83e"}
Sep 16 12:40:10.485908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.485821    2560 scope.go:117] "RemoveContainer" containerID="ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d"
Sep 16 12:40:10.495609 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.495596    2560 scope.go:117] "RemoveContainer" containerID="ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d"
Sep 16 12:40:10.495960 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:40:10.495938    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d\": container with ID starting with ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d not found: ID does not exist" containerID="ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d"
Sep 16 12:40:10.496028 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.495966    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d"} err="failed to get container status \"ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d\": rpc error: code = NotFound desc = could not find container \"ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d\": container with ID starting with ae1e58220e377bcdcccb909e216d07a71555fcf32536fef8e64d9cc763ab330d not found: ID does not exist"
Sep 16 12:40:10.503839 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.503816    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"]
Sep 16 12:40:10.516218 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:10.516199    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-869204-74b9b4669d-lqjf7"]
Sep 16 12:40:12.325097 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:12.325062    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a930391e-0efa-41c8-830e-67ea08ef65a2" path="/var/lib/kubelet/pods/a930391e-0efa-41c8-830e-67ea08ef65a2/volumes"
Sep 16 12:40:15.170737 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:15.168172    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"]
Sep 16 12:40:24.880534 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:24.880501    2560 scope.go:117] "RemoveContainer" containerID="140e0d15376a8e9b689ef43cc716d11fd29e66036bd63b7c6e57ebc20ed1b7bc"
Sep 16 12:40:30.193468 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.193432    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz" podUID="9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" containerName="main" containerID="cri-o://fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921" gracePeriod=45
Sep 16 12:40:30.243668 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:40:30.243640    2560 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ab272ca_f723_4d96_9d91_ebc9fd20bdd4.slice/crio-conmon-fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 12:40:30.454454 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.454400    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:40:30.505547 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.505520    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-model-cache\") pod \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") "
Sep 16 12:40:30.505719 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.505613    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tls-certs\") pod \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") "
Sep 16 12:40:30.505719 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.505645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-dshm\") pod \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") "
Sep 16 12:40:30.505719 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.505667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-home\") pod \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") "
Sep 16 12:40:30.505719 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.505688    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tmp-dir\") pod \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") "
Sep 16 12:40:30.505958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.505736    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p6vd\" (UniqueName: \"kubernetes.io/projected/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-kube-api-access-9p6vd\") pod \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\" (UID: \"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4\") "
Sep 16 12:40:30.505958 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.505812    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" (UID: "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:40:30.506060 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.506034    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:30.506239 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.506207    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-home" (OuterVolumeSpecName: "home") pod "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" (UID: "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:40:30.506318 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.506256    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" (UID: "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:40:30.508321 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.508235    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-kube-api-access-9p6vd" (OuterVolumeSpecName: "kube-api-access-9p6vd") pod "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" (UID: "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4"). InnerVolumeSpecName "kube-api-access-9p6vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:40:30.508321 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.508263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" (UID: "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:40:30.508512 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.508383    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-dshm" (OuterVolumeSpecName: "dshm") pod "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" (UID: "9ab272ca-f723-4d96-9d91-ebc9fd20bdd4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:40:30.596416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.596386    2560 generic.go:356] "Generic (PLEG): container finished" podID="9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" containerID="fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921" exitCode=0
Sep 16 12:40:30.596560 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.596434    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"
Sep 16 12:40:30.596560 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.596479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz" event={"ID":"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4","Type":"ContainerDied","Data":"fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921"}
Sep 16 12:40:30.596560 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.596523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz" event={"ID":"9ab272ca-f723-4d96-9d91-ebc9fd20bdd4","Type":"ContainerDied","Data":"9a1bdc9e94465e3136f5e8f2cb745d5ed1ae108f51ece22bd642e826957542a0"}
Sep 16 12:40:30.596560 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.596547    2560 scope.go:117] "RemoveContainer" containerID="fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921"
Sep 16 12:40:30.605493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.605476    2560 scope.go:117] "RemoveContainer" containerID="fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921"
Sep 16 12:40:30.605773 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:40:30.605751    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921\": container with ID starting with fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921 not found: ID does not exist" containerID="fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921"
Sep 16 12:40:30.605855 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.605781    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921"} err="failed to get container status \"fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921\": rpc error: code = NotFound desc = could not find container \"fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921\": container with ID starting with fdca08c21a7398086a2c13cbf8af8412aa980eedbd234578d865c5de6c67b921 not found: ID does not exist"
Sep 16 12:40:30.607196 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.607178    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:30.607196 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.607197    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:30.607313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.607205    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:30.607313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.607214    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:30.607313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.607230    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p6vd\" (UniqueName: \"kubernetes.io/projected/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4-kube-api-access-9p6vd\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:40:30.617484 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.617462    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"]
Sep 16 12:40:30.621585 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:30.621567    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-869204/ipp-route-869204-kserve-77486654f5-lgxlz"]
Sep 16 12:40:32.324433 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:32.324386    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" path="/var/lib/kubelet/pods/9ab272ca-f723-4d96-9d91-ebc9fd20bdd4/volumes"
Sep 16 12:40:59.396387 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396307    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"]
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396704    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a930391e-0efa-41c8-830e-67ea08ef65a2" containerName="payload-pre-processing"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396715    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a930391e-0efa-41c8-830e-67ea08ef65a2" containerName="payload-pre-processing"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396724    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" containerName="main"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396729    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" containerName="main"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396734    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a98157-405a-41cf-877b-ef4f13ed00ce" containerName="revoke-keys"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396740    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a98157-405a-41cf-877b-ef4f13ed00ce" containerName="revoke-keys"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396811    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a98157-405a-41cf-877b-ef4f13ed00ce" containerName="revoke-keys"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396821    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a930391e-0efa-41c8-830e-67ea08ef65a2" containerName="payload-pre-processing"
Sep 16 12:40:59.396860 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.396826    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab272ca-f723-4d96-9d91-ebc9fd20bdd4" containerName="main"
Sep 16 12:40:59.399646 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.399628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"
Sep 16 12:40:59.402385 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.402364    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:40:59.402492 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.402364    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:40:59.412326 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.412305    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"]
Sep 16 12:40:59.532276 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.532249    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93282379-196b-45af-aebe-ec057c4b7a6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h\" (UID: \"93282379-196b-45af-aebe-ec057c4b7a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"
Sep 16 12:40:59.633272 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.633245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93282379-196b-45af-aebe-ec057c4b7a6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h\" (UID: \"93282379-196b-45af-aebe-ec057c4b7a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"
Sep 16 12:40:59.633861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.633845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93282379-196b-45af-aebe-ec057c4b7a6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h\" (UID: \"93282379-196b-45af-aebe-ec057c4b7a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"
Sep 16 12:40:59.711131 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.711081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"
Sep 16 12:40:59.983915 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:40:59.983891    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"]
Sep 16 12:40:59.986029 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:40:59.985994    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93282379_196b_45af_aebe_ec057c4b7a6e.slice/crio-63c141bd871cac1fbdd82788c4d5d83ea84931a814e92be6f00c268eec73c91f WatchSource:0}: Error finding container 63c141bd871cac1fbdd82788c4d5d83ea84931a814e92be6f00c268eec73c91f: Status 404 returned error can't find the container with id 63c141bd871cac1fbdd82788c4d5d83ea84931a814e92be6f00c268eec73c91f
Sep 16 12:41:00.754962 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:00.754930    2560 generic.go:356] "Generic (PLEG): container finished" podID="93282379-196b-45af-aebe-ec057c4b7a6e" containerID="d22e63ac31efd7d022e854161da8493a791be1b5ab2ad29b6e7d107b7afff971" exitCode=0
Sep 16 12:41:00.755293 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:00.754967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h" event={"ID":"93282379-196b-45af-aebe-ec057c4b7a6e","Type":"ContainerDied","Data":"d22e63ac31efd7d022e854161da8493a791be1b5ab2ad29b6e7d107b7afff971"}
Sep 16 12:41:00.755293 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:00.755051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h" event={"ID":"93282379-196b-45af-aebe-ec057c4b7a6e","Type":"ContainerStarted","Data":"63c141bd871cac1fbdd82788c4d5d83ea84931a814e92be6f00c268eec73c91f"}
Sep 16 12:41:01.890862 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:01.890832    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"
Sep 16 12:41:01.953060 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:01.953033    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93282379-196b-45af-aebe-ec057c4b7a6e-maas-api-service-ca\") pod \"93282379-196b-45af-aebe-ec057c4b7a6e\" (UID: \"93282379-196b-45af-aebe-ec057c4b7a6e\") "
Sep 16 12:41:01.953343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:01.953321    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93282379-196b-45af-aebe-ec057c4b7a6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93282379-196b-45af-aebe-ec057c4b7a6e" (UID: "93282379-196b-45af-aebe-ec057c4b7a6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:02.054486 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:02.054422    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93282379-196b-45af-aebe-ec057c4b7a6e-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:02.768038 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:02.768012    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"
Sep 16 12:41:02.768196 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:02.768012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h" event={"ID":"93282379-196b-45af-aebe-ec057c4b7a6e","Type":"ContainerDied","Data":"63c141bd871cac1fbdd82788c4d5d83ea84931a814e92be6f00c268eec73c91f"}
Sep 16 12:41:02.768196 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:02.768139    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c141bd871cac1fbdd82788c4d5d83ea84931a814e92be6f00c268eec73c91f"
Sep 16 12:41:10.211163 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.211131    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"]
Sep 16 12:41:10.211645 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.211327    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2" podUID="f6e75adf-f968-4d5c-adc1-b9d78ab90520" containerName="payload-processing" containerID="cri-o://7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a" gracePeriod=30
Sep 16 12:41:10.217934 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.217530    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"]
Sep 16 12:41:10.217934 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.217781    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df" podUID="aa718987-1722-4f84-af4c-d048b7eedda8" containerName="payload-pre-processing" containerID="cri-o://73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750" gracePeriod=30
Sep 16 12:41:10.448815 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.448763    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"]
Sep 16 12:41:10.451980 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.451958    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-89b7ca-xj98h"]
Sep 16 12:41:10.517412 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.517386    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:41:10.521139 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.521113    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:41:10.620991 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.620963    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e75adf-f968-4d5c-adc1-b9d78ab90520-plugins-config-volume\") pod \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") "
Sep 16 12:41:10.621185 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.621017    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-724d4\" (UniqueName: \"kubernetes.io/projected/f6e75adf-f968-4d5c-adc1-b9d78ab90520-kube-api-access-724d4\") pod \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\" (UID: \"f6e75adf-f968-4d5c-adc1-b9d78ab90520\") "
Sep 16 12:41:10.621185 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.621053    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa718987-1722-4f84-af4c-d048b7eedda8-plugins-config-volume\") pod \"aa718987-1722-4f84-af4c-d048b7eedda8\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") "
Sep 16 12:41:10.621185 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.621100    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbmnq\" (UniqueName: \"kubernetes.io/projected/aa718987-1722-4f84-af4c-d048b7eedda8-kube-api-access-rbmnq\") pod \"aa718987-1722-4f84-af4c-d048b7eedda8\" (UID: \"aa718987-1722-4f84-af4c-d048b7eedda8\") "
Sep 16 12:41:10.621401 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.621374    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e75adf-f968-4d5c-adc1-b9d78ab90520-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6e75adf-f968-4d5c-adc1-b9d78ab90520" (UID: "f6e75adf-f968-4d5c-adc1-b9d78ab90520"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:10.621484 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.621458    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa718987-1722-4f84-af4c-d048b7eedda8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa718987-1722-4f84-af4c-d048b7eedda8" (UID: "aa718987-1722-4f84-af4c-d048b7eedda8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:10.623368 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.623343    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e75adf-f968-4d5c-adc1-b9d78ab90520-kube-api-access-724d4" (OuterVolumeSpecName: "kube-api-access-724d4") pod "f6e75adf-f968-4d5c-adc1-b9d78ab90520" (UID: "f6e75adf-f968-4d5c-adc1-b9d78ab90520"). InnerVolumeSpecName "kube-api-access-724d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:10.623481 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.623446    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa718987-1722-4f84-af4c-d048b7eedda8-kube-api-access-rbmnq" (OuterVolumeSpecName: "kube-api-access-rbmnq") pod "aa718987-1722-4f84-af4c-d048b7eedda8" (UID: "aa718987-1722-4f84-af4c-d048b7eedda8"). InnerVolumeSpecName "kube-api-access-rbmnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:10.721774 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.721753    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e75adf-f968-4d5c-adc1-b9d78ab90520-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.721774 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.721773    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-724d4\" (UniqueName: \"kubernetes.io/projected/f6e75adf-f968-4d5c-adc1-b9d78ab90520-kube-api-access-724d4\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.721915 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.721782    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa718987-1722-4f84-af4c-d048b7eedda8-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.721915 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.721812    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmnq\" (UniqueName: \"kubernetes.io/projected/aa718987-1722-4f84-af4c-d048b7eedda8-kube-api-access-rbmnq\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.815421 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.815363    2560 generic.go:356] "Generic (PLEG): container finished" podID="aa718987-1722-4f84-af4c-d048b7eedda8" containerID="73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750" exitCode=0
Sep 16 12:41:10.815421 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.815402    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"
Sep 16 12:41:10.815554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.815419    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df" event={"ID":"aa718987-1722-4f84-af4c-d048b7eedda8","Type":"ContainerDied","Data":"73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750"}
Sep 16 12:41:10.815554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.815481    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df" event={"ID":"aa718987-1722-4f84-af4c-d048b7eedda8","Type":"ContainerDied","Data":"e2c3f88d050ff17001c36303d0ef3bc4fe9008edcdc20c523b2ce3d77216f32d"}
Sep 16 12:41:10.815554 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.815513    2560 scope.go:117] "RemoveContainer" containerID="73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750"
Sep 16 12:41:10.817289 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.817266    2560 generic.go:356] "Generic (PLEG): container finished" podID="f6e75adf-f968-4d5c-adc1-b9d78ab90520" containerID="7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a" exitCode=0
Sep 16 12:41:10.817376 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.817316    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"
Sep 16 12:41:10.817376 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.817330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2" event={"ID":"f6e75adf-f968-4d5c-adc1-b9d78ab90520","Type":"ContainerDied","Data":"7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a"}
Sep 16 12:41:10.817376 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.817368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2" event={"ID":"f6e75adf-f968-4d5c-adc1-b9d78ab90520","Type":"ContainerDied","Data":"f6c019f333949f4fe414faf03a8a3e1f3ed30f6be4b86061e436571bb33045c9"}
Sep 16 12:41:10.828488 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.828461    2560 scope.go:117] "RemoveContainer" containerID="73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750"
Sep 16 12:41:10.828907 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:41:10.828876    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750\": container with ID starting with 73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750 not found: ID does not exist" containerID="73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750"
Sep 16 12:41:10.829016 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.828919    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750"} err="failed to get container status \"73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750\": rpc error: code = NotFound desc = could not find container \"73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750\": container with ID starting with 73a447106f387ad8d337f4f0971c8a18e82714a61c9e76d1c8f51cd13d447750 not found: ID does not exist"
Sep 16 12:41:10.829016 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.828944    2560 scope.go:117] "RemoveContainer" containerID="7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a"
Sep 16 12:41:10.838826 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.838776    2560 scope.go:117] "RemoveContainer" containerID="7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a"
Sep 16 12:41:10.839099 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:41:10.839081    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a\": container with ID starting with 7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a not found: ID does not exist" containerID="7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a"
Sep 16 12:41:10.839143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.839106    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a"} err="failed to get container status \"7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a\": rpc error: code = NotFound desc = could not find container \"7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a\": container with ID starting with 7d94291d03048733fc5230989cd41c4410032b66e812fe2d0bd0a8ac26ff677a not found: ID does not exist"
Sep 16 12:41:10.863645 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.863618    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"]
Sep 16 12:41:10.869491 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.869471    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-89b7ca-6fcdcdbd55-jqxw2"]
Sep 16 12:41:10.884288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.884265    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"]
Sep 16 12:41:10.888842 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:10.888821    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-89b7ca-bb46c547d-pn2df"]
Sep 16 12:41:12.324996 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:12.324965    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93282379-196b-45af-aebe-ec057c4b7a6e" path="/var/lib/kubelet/pods/93282379-196b-45af-aebe-ec057c4b7a6e/volumes"
Sep 16 12:41:12.325354 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:12.325310    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa718987-1722-4f84-af4c-d048b7eedda8" path="/var/lib/kubelet/pods/aa718987-1722-4f84-af4c-d048b7eedda8/volumes"
Sep 16 12:41:12.325661 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:12.325646    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e75adf-f968-4d5c-adc1-b9d78ab90520" path="/var/lib/kubelet/pods/f6e75adf-f968-4d5c-adc1-b9d78ab90520/volumes"
Sep 16 12:41:17.387294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387263    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"]
Sep 16 12:41:17.387662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387620    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa718987-1722-4f84-af4c-d048b7eedda8" containerName="payload-pre-processing"
Sep 16 12:41:17.387662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387630    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa718987-1722-4f84-af4c-d048b7eedda8" containerName="payload-pre-processing"
Sep 16 12:41:17.387662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387647    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93282379-196b-45af-aebe-ec057c4b7a6e" containerName="revoke-keys"
Sep 16 12:41:17.387662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387653    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="93282379-196b-45af-aebe-ec057c4b7a6e" containerName="revoke-keys"
Sep 16 12:41:17.387662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387659    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e75adf-f968-4d5c-adc1-b9d78ab90520" containerName="payload-processing"
Sep 16 12:41:17.387662 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387665    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e75adf-f968-4d5c-adc1-b9d78ab90520" containerName="payload-processing"
Sep 16 12:41:17.387984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387733    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="93282379-196b-45af-aebe-ec057c4b7a6e" containerName="revoke-keys"
Sep 16 12:41:17.387984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387742    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e75adf-f968-4d5c-adc1-b9d78ab90520" containerName="payload-processing"
Sep 16 12:41:17.387984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.387750    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa718987-1722-4f84-af4c-d048b7eedda8" containerName="payload-pre-processing"
Sep 16 12:41:17.392192 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.392171    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"
Sep 16 12:41:17.394953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.394933    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:41:17.395085 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.394933    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:41:17.399207 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.399185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"]
Sep 16 12:41:17.478177 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.478148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969eb0da-843b-4fa0-b1fb-80dd3e728240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s\" (UID: \"969eb0da-843b-4fa0-b1fb-80dd3e728240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"
Sep 16 12:41:17.579590 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.579557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969eb0da-843b-4fa0-b1fb-80dd3e728240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s\" (UID: \"969eb0da-843b-4fa0-b1fb-80dd3e728240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"
Sep 16 12:41:17.580246 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.580217    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969eb0da-843b-4fa0-b1fb-80dd3e728240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s\" (UID: \"969eb0da-843b-4fa0-b1fb-80dd3e728240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"
Sep 16 12:41:17.704852 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.704759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"
Sep 16 12:41:17.976711 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:17.976679    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"]
Sep 16 12:41:17.979077 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:41:17.979049    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969eb0da_843b_4fa0_b1fb_80dd3e728240.slice/crio-56ab789da31eb77f9fe8a91c93f353b7dec7c99da1ba7c47635bb47128a20245 WatchSource:0}: Error finding container 56ab789da31eb77f9fe8a91c93f353b7dec7c99da1ba7c47635bb47128a20245: Status 404 returned error can't find the container with id 56ab789da31eb77f9fe8a91c93f353b7dec7c99da1ba7c47635bb47128a20245
Sep 16 12:41:18.862586 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:18.862548    2560 generic.go:356] "Generic (PLEG): container finished" podID="969eb0da-843b-4fa0-b1fb-80dd3e728240" containerID="ee88c1878e9e99954ff26e0739466b880b98a64593574746a1f81778f02febdf" exitCode=0
Sep 16 12:41:18.862945 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:18.862603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s" event={"ID":"969eb0da-843b-4fa0-b1fb-80dd3e728240","Type":"ContainerDied","Data":"ee88c1878e9e99954ff26e0739466b880b98a64593574746a1f81778f02febdf"}
Sep 16 12:41:18.862945 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:18.862657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s" event={"ID":"969eb0da-843b-4fa0-b1fb-80dd3e728240","Type":"ContainerStarted","Data":"56ab789da31eb77f9fe8a91c93f353b7dec7c99da1ba7c47635bb47128a20245"}
Sep 16 12:41:20.006075 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:20.006048    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"
Sep 16 12:41:20.098559 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:20.098534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969eb0da-843b-4fa0-b1fb-80dd3e728240-maas-api-service-ca\") pod \"969eb0da-843b-4fa0-b1fb-80dd3e728240\" (UID: \"969eb0da-843b-4fa0-b1fb-80dd3e728240\") "
Sep 16 12:41:20.098885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:20.098865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969eb0da-843b-4fa0-b1fb-80dd3e728240-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "969eb0da-843b-4fa0-b1fb-80dd3e728240" (UID: "969eb0da-843b-4fa0-b1fb-80dd3e728240"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:20.199938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:20.199868    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969eb0da-843b-4fa0-b1fb-80dd3e728240-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:20.873771 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:20.873733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s" event={"ID":"969eb0da-843b-4fa0-b1fb-80dd3e728240","Type":"ContainerDied","Data":"56ab789da31eb77f9fe8a91c93f353b7dec7c99da1ba7c47635bb47128a20245"}
Sep 16 12:41:20.873771 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:20.873764    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"
Sep 16 12:41:20.873983 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:20.873803    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56ab789da31eb77f9fe8a91c93f353b7dec7c99da1ba7c47635bb47128a20245"
Sep 16 12:41:27.954232 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:27.954182    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"]
Sep 16 12:41:27.954690 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:27.954420    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg" podUID="f070e4b1-f730-4fbc-bbff-7c490ab7f230" containerName="payload-processing" containerID="cri-o://986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8" gracePeriod=30
Sep 16 12:41:27.959493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:27.959466    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"]
Sep 16 12:41:27.959897 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:27.959853    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7" podUID="e6357ed4-8477-4a9f-85d5-da8d3c0fab48" containerName="payload-pre-processing" containerID="cri-o://4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423" gracePeriod=30
Sep 16 12:41:28.130133 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.130093    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"]
Sep 16 12:41:28.133378 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.133349    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a9008-5jd9s"]
Sep 16 12:41:28.239475 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.239453    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:41:28.242516 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.242496    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:41:28.325292 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.325265    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969eb0da-843b-4fa0-b1fb-80dd3e728240" path="/var/lib/kubelet/pods/969eb0da-843b-4fa0-b1fb-80dd3e728240/volumes"
Sep 16 12:41:28.369692 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.369663    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxtng\" (UniqueName: \"kubernetes.io/projected/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-kube-api-access-mxtng\") pod \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") "
Sep 16 12:41:28.369836 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.369715    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-plugins-config-volume\") pod \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\" (UID: \"e6357ed4-8477-4a9f-85d5-da8d3c0fab48\") "
Sep 16 12:41:28.369836 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.369775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f070e4b1-f730-4fbc-bbff-7c490ab7f230-plugins-config-volume\") pod \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") "
Sep 16 12:41:28.369938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.369847    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr4vc\" (UniqueName: \"kubernetes.io/projected/f070e4b1-f730-4fbc-bbff-7c490ab7f230-kube-api-access-wr4vc\") pod \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\" (UID: \"f070e4b1-f730-4fbc-bbff-7c490ab7f230\") "
Sep 16 12:41:28.370189 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.370151    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6357ed4-8477-4a9f-85d5-da8d3c0fab48" (UID: "e6357ed4-8477-4a9f-85d5-da8d3c0fab48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:28.370189 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.370150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f070e4b1-f730-4fbc-bbff-7c490ab7f230-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f070e4b1-f730-4fbc-bbff-7c490ab7f230" (UID: "f070e4b1-f730-4fbc-bbff-7c490ab7f230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:28.372002 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.371976    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f070e4b1-f730-4fbc-bbff-7c490ab7f230-kube-api-access-wr4vc" (OuterVolumeSpecName: "kube-api-access-wr4vc") pod "f070e4b1-f730-4fbc-bbff-7c490ab7f230" (UID: "f070e4b1-f730-4fbc-bbff-7c490ab7f230"). InnerVolumeSpecName "kube-api-access-wr4vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:28.372148 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.372130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-kube-api-access-mxtng" (OuterVolumeSpecName: "kube-api-access-mxtng") pod "e6357ed4-8477-4a9f-85d5-da8d3c0fab48" (UID: "e6357ed4-8477-4a9f-85d5-da8d3c0fab48"). InnerVolumeSpecName "kube-api-access-mxtng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:28.471305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.470701    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtng\" (UniqueName: \"kubernetes.io/projected/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-kube-api-access-mxtng\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.471305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.470730    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6357ed4-8477-4a9f-85d5-da8d3c0fab48-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.471305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.470744    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f070e4b1-f730-4fbc-bbff-7c490ab7f230-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.471305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.470757    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4vc\" (UniqueName: \"kubernetes.io/projected/f070e4b1-f730-4fbc-bbff-7c490ab7f230-kube-api-access-wr4vc\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.917740 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.917702    2560 generic.go:356] "Generic (PLEG): container finished" podID="f070e4b1-f730-4fbc-bbff-7c490ab7f230" containerID="986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8" exitCode=0
Sep 16 12:41:28.917896 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.917764    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"
Sep 16 12:41:28.917896 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.917781    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg" event={"ID":"f070e4b1-f730-4fbc-bbff-7c490ab7f230","Type":"ContainerDied","Data":"986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8"}
Sep 16 12:41:28.917896 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.917870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg" event={"ID":"f070e4b1-f730-4fbc-bbff-7c490ab7f230","Type":"ContainerDied","Data":"26a3dd7f9d00f2a5d3eca80666e5df071aaf56b8fce1b5dd24784f1f6f6d3975"}
Sep 16 12:41:28.918081 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.917894    2560 scope.go:117] "RemoveContainer" containerID="986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8"
Sep 16 12:41:28.919407 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.919384    2560 generic.go:356] "Generic (PLEG): container finished" podID="e6357ed4-8477-4a9f-85d5-da8d3c0fab48" containerID="4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423" exitCode=0
Sep 16 12:41:28.919526 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.919413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7" event={"ID":"e6357ed4-8477-4a9f-85d5-da8d3c0fab48","Type":"ContainerDied","Data":"4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423"}
Sep 16 12:41:28.919526 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.919442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7" event={"ID":"e6357ed4-8477-4a9f-85d5-da8d3c0fab48","Type":"ContainerDied","Data":"f8cb93c03479b5f88f7b321e8d42a3d31442501c0811c9b1a226cd314cb09ec2"}
Sep 16 12:41:28.919526 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.919439    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"
Sep 16 12:41:28.929016 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.929000    2560 scope.go:117] "RemoveContainer" containerID="986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8"
Sep 16 12:41:28.929365 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:41:28.929339    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8\": container with ID starting with 986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8 not found: ID does not exist" containerID="986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8"
Sep 16 12:41:28.929453 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.929376    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8"} err="failed to get container status \"986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8\": rpc error: code = NotFound desc = could not find container \"986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8\": container with ID starting with 986a8650d00c74a3e991b45c24e49f3ce6dd867721fbd066dbadbc15981239f8 not found: ID does not exist"
Sep 16 12:41:28.929453 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.929400    2560 scope.go:117] "RemoveContainer" containerID="4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423"
Sep 16 12:41:28.938335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.938318    2560 scope.go:117] "RemoveContainer" containerID="4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423"
Sep 16 12:41:28.938618 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:41:28.938601    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423\": container with ID starting with 4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423 not found: ID does not exist" containerID="4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423"
Sep 16 12:41:28.938679 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.938624    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423"} err="failed to get container status \"4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423\": rpc error: code = NotFound desc = could not find container \"4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423\": container with ID starting with 4c9013cb7005ac2b06066d9c0335fc2c1fe0909f6d7be16202f35e552a5db423 not found: ID does not exist"
Sep 16 12:41:28.945283 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.945262    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"]
Sep 16 12:41:28.949227 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.949208    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0a9008-54dfc4d547-v2wfg"]
Sep 16 12:41:28.964312 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.964283    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"]
Sep 16 12:41:28.967334 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:28.967314    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a9008-6d8479cf8-6zkz7"]
Sep 16 12:41:30.324720 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:30.324684    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6357ed4-8477-4a9f-85d5-da8d3c0fab48" path="/var/lib/kubelet/pods/e6357ed4-8477-4a9f-85d5-da8d3c0fab48/volumes"
Sep 16 12:41:30.325237 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:30.325220    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f070e4b1-f730-4fbc-bbff-7c490ab7f230" path="/var/lib/kubelet/pods/f070e4b1-f730-4fbc-bbff-7c490ab7f230/volumes"
Sep 16 12:41:35.417858 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.417823    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"]
Sep 16 12:41:35.418305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418222    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969eb0da-843b-4fa0-b1fb-80dd3e728240" containerName="revoke-keys"
Sep 16 12:41:35.418305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418236    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="969eb0da-843b-4fa0-b1fb-80dd3e728240" containerName="revoke-keys"
Sep 16 12:41:35.418305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418247    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6357ed4-8477-4a9f-85d5-da8d3c0fab48" containerName="payload-pre-processing"
Sep 16 12:41:35.418305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418252    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6357ed4-8477-4a9f-85d5-da8d3c0fab48" containerName="payload-pre-processing"
Sep 16 12:41:35.418305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418269    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f070e4b1-f730-4fbc-bbff-7c490ab7f230" containerName="payload-processing"
Sep 16 12:41:35.418305 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418275    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f070e4b1-f730-4fbc-bbff-7c490ab7f230" containerName="payload-processing"
Sep 16 12:41:35.418528 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418340    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f070e4b1-f730-4fbc-bbff-7c490ab7f230" containerName="payload-processing"
Sep 16 12:41:35.418528 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418349    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="969eb0da-843b-4fa0-b1fb-80dd3e728240" containerName="revoke-keys"
Sep 16 12:41:35.418528 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.418358    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6357ed4-8477-4a9f-85d5-da8d3c0fab48" containerName="payload-pre-processing"
Sep 16 12:41:35.422335 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.422319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"
Sep 16 12:41:35.429600 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.429578    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:41:35.429867 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.429850    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:41:35.438068 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.438045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"]
Sep 16 12:41:35.528165 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.528139    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7c6ef0-731d-4f9b-8ced-64e6d267837a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n\" (UID: \"ca7c6ef0-731d-4f9b-8ced-64e6d267837a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"
Sep 16 12:41:35.629320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.629291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7c6ef0-731d-4f9b-8ced-64e6d267837a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n\" (UID: \"ca7c6ef0-731d-4f9b-8ced-64e6d267837a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"
Sep 16 12:41:35.629888 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.629872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7c6ef0-731d-4f9b-8ced-64e6d267837a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n\" (UID: \"ca7c6ef0-731d-4f9b-8ced-64e6d267837a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"
Sep 16 12:41:35.733076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:35.733011    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"
Sep 16 12:41:36.007159 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:36.007082    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"]
Sep 16 12:41:36.009612 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:41:36.009587    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7c6ef0_731d_4f9b_8ced_64e6d267837a.slice/crio-29fc53a74fb47f01421eb90564cdb85af3aa76c4f84e1cb36547787dc29cd00f WatchSource:0}: Error finding container 29fc53a74fb47f01421eb90564cdb85af3aa76c4f84e1cb36547787dc29cd00f: Status 404 returned error can't find the container with id 29fc53a74fb47f01421eb90564cdb85af3aa76c4f84e1cb36547787dc29cd00f
Sep 16 12:41:36.967254 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:36.967177    2560 generic.go:356] "Generic (PLEG): container finished" podID="ca7c6ef0-731d-4f9b-8ced-64e6d267837a" containerID="76d1c188a8cccd9dff13d6ba97e4d77653eb941844f20a2e85d485a213d116d3" exitCode=0
Sep 16 12:41:36.967569 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:36.967258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n" event={"ID":"ca7c6ef0-731d-4f9b-8ced-64e6d267837a","Type":"ContainerDied","Data":"76d1c188a8cccd9dff13d6ba97e4d77653eb941844f20a2e85d485a213d116d3"}
Sep 16 12:41:36.967569 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:36.967303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n" event={"ID":"ca7c6ef0-731d-4f9b-8ced-64e6d267837a","Type":"ContainerStarted","Data":"29fc53a74fb47f01421eb90564cdb85af3aa76c4f84e1cb36547787dc29cd00f"}
Sep 16 12:41:38.101024 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:38.101002    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"
Sep 16 12:41:38.151752 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:38.151725    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7c6ef0-731d-4f9b-8ced-64e6d267837a-maas-api-service-ca\") pod \"ca7c6ef0-731d-4f9b-8ced-64e6d267837a\" (UID: \"ca7c6ef0-731d-4f9b-8ced-64e6d267837a\") "
Sep 16 12:41:38.152084 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:38.152062    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7c6ef0-731d-4f9b-8ced-64e6d267837a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca7c6ef0-731d-4f9b-8ced-64e6d267837a" (UID: "ca7c6ef0-731d-4f9b-8ced-64e6d267837a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:38.252513 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:38.252447    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca7c6ef0-731d-4f9b-8ced-64e6d267837a-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:38.984862 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:38.984833    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"
Sep 16 12:41:38.985050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:38.984874    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n" event={"ID":"ca7c6ef0-731d-4f9b-8ced-64e6d267837a","Type":"ContainerDied","Data":"29fc53a74fb47f01421eb90564cdb85af3aa76c4f84e1cb36547787dc29cd00f"}
Sep 16 12:41:38.985050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:38.984939    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29fc53a74fb47f01421eb90564cdb85af3aa76c4f84e1cb36547787dc29cd00f"
Sep 16 12:41:46.001637 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.001605    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"]
Sep 16 12:41:46.006463 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.006435    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e94e5-29826030-txwvl"]
Sep 16 12:41:46.033133 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.033100    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"]
Sep 16 12:41:46.033381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.033353    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk" podUID="9947af79-2b99-44d7-930c-ca4cf744f706" containerName="payload-pre-processing" containerID="cri-o://1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53" gracePeriod=30
Sep 16 12:41:46.196133 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.196092    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"]
Sep 16 12:41:46.197831 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.197807    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e94e5-hc72n"]
Sep 16 12:41:46.287270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.287246    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:41:46.320345 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.320313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftzrr\" (UniqueName: \"kubernetes.io/projected/9947af79-2b99-44d7-930c-ca4cf744f706-kube-api-access-ftzrr\") pod \"9947af79-2b99-44d7-930c-ca4cf744f706\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") "
Sep 16 12:41:46.320345 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.320359    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9947af79-2b99-44d7-930c-ca4cf744f706-plugins-config-volume\") pod \"9947af79-2b99-44d7-930c-ca4cf744f706\" (UID: \"9947af79-2b99-44d7-930c-ca4cf744f706\") "
Sep 16 12:41:46.320938 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.320906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9947af79-2b99-44d7-930c-ca4cf744f706-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9947af79-2b99-44d7-930c-ca4cf744f706" (UID: "9947af79-2b99-44d7-930c-ca4cf744f706"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:46.321099 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.321047    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9947af79-2b99-44d7-930c-ca4cf744f706-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:46.322980 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.322953    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9947af79-2b99-44d7-930c-ca4cf744f706-kube-api-access-ftzrr" (OuterVolumeSpecName: "kube-api-access-ftzrr") pod "9947af79-2b99-44d7-930c-ca4cf744f706" (UID: "9947af79-2b99-44d7-930c-ca4cf744f706"). InnerVolumeSpecName "kube-api-access-ftzrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:46.325684 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.325654    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34407539-9364-4e97-b635-85acd0f42588" path="/var/lib/kubelet/pods/34407539-9364-4e97-b635-85acd0f42588/volumes"
Sep 16 12:41:46.326036 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.326018    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca7c6ef0-731d-4f9b-8ced-64e6d267837a" path="/var/lib/kubelet/pods/ca7c6ef0-731d-4f9b-8ced-64e6d267837a/volumes"
Sep 16 12:41:46.421450 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:46.421426    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzrr\" (UniqueName: \"kubernetes.io/projected/9947af79-2b99-44d7-930c-ca4cf744f706-kube-api-access-ftzrr\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:41:47.028328 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.028243    2560 generic.go:356] "Generic (PLEG): container finished" podID="9947af79-2b99-44d7-930c-ca4cf744f706" containerID="1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53" exitCode=0
Sep 16 12:41:47.028328 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.028289    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"
Sep 16 12:41:47.028885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.028288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk" event={"ID":"9947af79-2b99-44d7-930c-ca4cf744f706","Type":"ContainerDied","Data":"1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53"}
Sep 16 12:41:47.028885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.028404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk" event={"ID":"9947af79-2b99-44d7-930c-ca4cf744f706","Type":"ContainerDied","Data":"cc1179eb9b333ddfee46a4fdbf3a32d595deeb5ce0325f318779325e74eec61b"}
Sep 16 12:41:47.028885 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.028477    2560 scope.go:117] "RemoveContainer" containerID="1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53"
Sep 16 12:41:47.038127 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.037938    2560 scope.go:117] "RemoveContainer" containerID="1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53"
Sep 16 12:41:47.038236 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:41:47.038215    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53\": container with ID starting with 1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53 not found: ID does not exist" containerID="1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53"
Sep 16 12:41:47.038281 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.038244    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53"} err="failed to get container status \"1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53\": rpc error: code = NotFound desc = could not find container \"1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53\": container with ID starting with 1cd64cc2997c5d0cce35e8b04ae09db85aa5e5bc3a19d9f32bacd2f10a655e53 not found: ID does not exist"
Sep 16 12:41:47.047927 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.047907    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"]
Sep 16 12:41:47.055244 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:47.055214    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e94e5-7dd675bd54-bbkhk"]
Sep 16 12:41:48.324217 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:41:48.324183    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9947af79-2b99-44d7-930c-ca4cf744f706" path="/var/lib/kubelet/pods/9947af79-2b99-44d7-930c-ca4cf744f706/volumes"
Sep 16 12:42:24.932925 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:42:24.932852    2560 scope.go:117] "RemoveContainer" containerID="97b9076ba7083d20612be9d81bb33ac13574d04a6ad6994747b3542ab224fbb9"
Sep 16 12:43:24.496160 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:43:24.496124    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:43:24.503652 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:43:24.503630    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:44:55.692233 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:55.692195    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"]
Sep 16 12:44:55.693013 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:55.692478    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7" podUID="32025949-7bc3-437e-b7d5-728153f2a8a9" containerName="manager" containerID="cri-o://ff5e5a13511dbf98e80b7770bafd689d28045fe3b8419374da8f3f45ca956bf2" gracePeriod=10
Sep 16 12:44:56.991641 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:56.991576    2560 generic.go:356] "Generic (PLEG): container finished" podID="32025949-7bc3-437e-b7d5-728153f2a8a9" containerID="ff5e5a13511dbf98e80b7770bafd689d28045fe3b8419374da8f3f45ca956bf2" exitCode=0
Sep 16 12:44:56.991988 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:56.991651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7" event={"ID":"32025949-7bc3-437e-b7d5-728153f2a8a9","Type":"ContainerDied","Data":"ff5e5a13511dbf98e80b7770bafd689d28045fe3b8419374da8f3f45ca956bf2"}
Sep 16 12:44:57.326980 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.326959    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:44:57.518270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.518187    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bkm8\" (UniqueName: \"kubernetes.io/projected/32025949-7bc3-437e-b7d5-728153f2a8a9-kube-api-access-5bkm8\") pod \"32025949-7bc3-437e-b7d5-728153f2a8a9\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") "
Sep 16 12:44:57.518270 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.518268    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32025949-7bc3-437e-b7d5-728153f2a8a9-extensions-socket-volume\") pod \"32025949-7bc3-437e-b7d5-728153f2a8a9\" (UID: \"32025949-7bc3-437e-b7d5-728153f2a8a9\") "
Sep 16 12:44:57.518665 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.518639    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32025949-7bc3-437e-b7d5-728153f2a8a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "32025949-7bc3-437e-b7d5-728153f2a8a9" (UID: "32025949-7bc3-437e-b7d5-728153f2a8a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:44:57.520300 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.520278    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32025949-7bc3-437e-b7d5-728153f2a8a9-kube-api-access-5bkm8" (OuterVolumeSpecName: "kube-api-access-5bkm8") pod "32025949-7bc3-437e-b7d5-728153f2a8a9" (UID: "32025949-7bc3-437e-b7d5-728153f2a8a9"). InnerVolumeSpecName "kube-api-access-5bkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:44:57.619815 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.619764    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bkm8\" (UniqueName: \"kubernetes.io/projected/32025949-7bc3-437e-b7d5-728153f2a8a9-kube-api-access-5bkm8\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:44:57.619815 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.619814    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32025949-7bc3-437e-b7d5-728153f2a8a9-extensions-socket-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:44:57.998349 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.998317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7" event={"ID":"32025949-7bc3-437e-b7d5-728153f2a8a9","Type":"ContainerDied","Data":"0f57079991b20bc97dba1df70ec3208d1a52bf84b7aeff2ea2a93cf3bc87a9e8"}
Sep 16 12:44:57.998767 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.998354    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"
Sep 16 12:44:57.998767 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:57.998378    2560 scope.go:117] "RemoveContainer" containerID="ff5e5a13511dbf98e80b7770bafd689d28045fe3b8419374da8f3f45ca956bf2"
Sep 16 12:44:58.020252 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:58.020228    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"]
Sep 16 12:44:58.023956 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:58.023935    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-24cm7"]
Sep 16 12:44:58.324515 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:44:58.324459    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32025949-7bc3-437e-b7d5-728153f2a8a9" path="/var/lib/kubelet/pods/32025949-7bc3-437e-b7d5-728153f2a8a9/volumes"
Sep 16 12:45:00.135403 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135369    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c"]
Sep 16 12:45:00.135760 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135719    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9947af79-2b99-44d7-930c-ca4cf744f706" containerName="payload-pre-processing"
Sep 16 12:45:00.135760 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135730    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9947af79-2b99-44d7-930c-ca4cf744f706" containerName="payload-pre-processing"
Sep 16 12:45:00.135760 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135752    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32025949-7bc3-437e-b7d5-728153f2a8a9" containerName="manager"
Sep 16 12:45:00.135760 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135757    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32025949-7bc3-437e-b7d5-728153f2a8a9" containerName="manager"
Sep 16 12:45:00.135918 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135764    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca7c6ef0-731d-4f9b-8ced-64e6d267837a" containerName="revoke-keys"
Sep 16 12:45:00.135918 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135769    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca7c6ef0-731d-4f9b-8ced-64e6d267837a" containerName="revoke-keys"
Sep 16 12:45:00.135918 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135838    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca7c6ef0-731d-4f9b-8ced-64e6d267837a" containerName="revoke-keys"
Sep 16 12:45:00.135918 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135845    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="32025949-7bc3-437e-b7d5-728153f2a8a9" containerName="manager"
Sep 16 12:45:00.135918 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.135854    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9947af79-2b99-44d7-930c-ca4cf744f706" containerName="payload-pre-processing"
Sep 16 12:45:00.139975 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.139959    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c"
Sep 16 12:45:00.142329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.142313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:45:00.146160 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.146088    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c"]
Sep 16 12:45:00.151308 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.151290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c"
Sep 16 12:45:00.419492 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:00.419471    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c"]
Sep 16 12:45:00.421715 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:45:00.421675    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607ed29e_1f18_486f_bfd5_8c15b44c74c8.slice/crio-5ec4c11514a0577a5292154dbb9b6779fbeaa859015ec33506ab21b79075af38 WatchSource:0}: Error finding container 5ec4c11514a0577a5292154dbb9b6779fbeaa859015ec33506ab21b79075af38: Status 404 returned error can't find the container with id 5ec4c11514a0577a5292154dbb9b6779fbeaa859015ec33506ab21b79075af38
Sep 16 12:45:01.016839 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:01.016730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c" event={"ID":"607ed29e-1f18-486f-bfd5-8c15b44c74c8","Type":"ContainerStarted","Data":"f8275a36ed0a1e680ec2e41bde47d2d3be15de3425f365ee4cadccdecf9b79e7"}
Sep 16 12:45:01.016839 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:01.016819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c" event={"ID":"607ed29e-1f18-486f-bfd5-8c15b44c74c8","Type":"ContainerStarted","Data":"5ec4c11514a0577a5292154dbb9b6779fbeaa859015ec33506ab21b79075af38"}
Sep 16 12:45:01.032686 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:01.032639    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c" podStartSLOduration=1.032625609 podStartE2EDuration="1.032625609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:45:01.031091828 +0000 UTC m=+2497.166413798" watchObservedRunningTime="2026-09-16 12:45:01.032625609 +0000 UTC m=+2497.167947580"
Sep 16 12:45:02.024050 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:02.024013    2560 generic.go:356] "Generic (PLEG): container finished" podID="607ed29e-1f18-486f-bfd5-8c15b44c74c8" containerID="f8275a36ed0a1e680ec2e41bde47d2d3be15de3425f365ee4cadccdecf9b79e7" exitCode=0
Sep 16 12:45:02.024399 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:02.024085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c" event={"ID":"607ed29e-1f18-486f-bfd5-8c15b44c74c8","Type":"ContainerDied","Data":"f8275a36ed0a1e680ec2e41bde47d2d3be15de3425f365ee4cadccdecf9b79e7"}
Sep 16 12:45:03.158152 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:03.158129    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c"
Sep 16 12:45:04.036114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:04.036082    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c"
Sep 16 12:45:04.036288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:04.036117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826045-ns75c" event={"ID":"607ed29e-1f18-486f-bfd5-8c15b44c74c8","Type":"ContainerDied","Data":"5ec4c11514a0577a5292154dbb9b6779fbeaa859015ec33506ab21b79075af38"}
Sep 16 12:45:04.036288 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:45:04.036166    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec4c11514a0577a5292154dbb9b6779fbeaa859015ec33506ab21b79075af38"
Sep 16 12:46:24.990459 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:24.990425    2560 scope.go:117] "RemoveContainer" containerID="fd6c007cb79121cfd04bc86e6e15a04717d97b141732f87a018ef48636c66154"
Sep 16 12:46:25.000395 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:25.000359    2560 scope.go:117] "RemoveContainer" containerID="2d78778428407d3a2ca0b1aac793077b6808398f433f8b74623033168baeffea"
Sep 16 12:46:38.253151 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.253119    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"]
Sep 16 12:46:38.253606 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.253591    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="607ed29e-1f18-486f-bfd5-8c15b44c74c8" containerName="cleanup"
Sep 16 12:46:38.253671 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.253610    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="607ed29e-1f18-486f-bfd5-8c15b44c74c8" containerName="cleanup"
Sep 16 12:46:38.253710 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.253690    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="607ed29e-1f18-486f-bfd5-8c15b44c74c8" containerName="cleanup"
Sep 16 12:46:38.256444 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.256429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.259737 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.259715    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5c14b3f0-kserve-self-signed-certs\""
Sep 16 12:46:38.266169 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.266148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tmp-dir\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.266275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.266177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tls-certs\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.266275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.266194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-dshm\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.266406 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.266329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnwtq\" (UniqueName: \"kubernetes.io/projected/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-kube-api-access-pnwtq\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.266464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.266409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-model-cache\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.266464 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.266439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-home\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.273533 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.273513    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"]
Sep 16 12:46:38.367542 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.367508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwtq\" (UniqueName: \"kubernetes.io/projected/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-kube-api-access-pnwtq\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.367688 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.367566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-model-cache\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.367688 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.367586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-home\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.367688 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.367609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tmp-dir\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.367688 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.367632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tls-certs\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.367688 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.367654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-dshm\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.368063 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.368036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tmp-dir\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.368147 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.368080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-home\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.368207 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.368183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-model-cache\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.370206 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.370185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-dshm\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.370328 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.370313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tls-certs\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.380908 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.380883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwtq\" (UniqueName: \"kubernetes.io/projected/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-kube-api-access-pnwtq\") pod \"e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") " pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.565881 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.565823    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:38.838105 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:38.838081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"]
Sep 16 12:46:38.840663 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:46:38.840642    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbedb1bfa_13ef_4141_93e5_fed6cd91cb92.slice/crio-3ceebdaf3c551f1a8ce669556bc00c2461db43e3e2fc8b1ea6ce18b4f4c10578 WatchSource:0}: Error finding container 3ceebdaf3c551f1a8ce669556bc00c2461db43e3e2fc8b1ea6ce18b4f4c10578: Status 404 returned error can't find the container with id 3ceebdaf3c551f1a8ce669556bc00c2461db43e3e2fc8b1ea6ce18b4f4c10578
Sep 16 12:46:39.516989 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:39.516956    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx" event={"ID":"bedb1bfa-13ef-4141-93e5-fed6cd91cb92","Type":"ContainerStarted","Data":"d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce"}
Sep 16 12:46:39.516989 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:39.516992    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx" event={"ID":"bedb1bfa-13ef-4141-93e5-fed6cd91cb92","Type":"ContainerStarted","Data":"3ceebdaf3c551f1a8ce669556bc00c2461db43e3e2fc8b1ea6ce18b4f4c10578"}
Sep 16 12:46:39.535114 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:39.535065    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx" podStartSLOduration=1.535050772 podStartE2EDuration="1.535050772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:39.534339913 +0000 UTC m=+2595.669661885" watchObservedRunningTime="2026-09-16 12:46:39.535050772 +0000 UTC m=+2595.670372743"
Sep 16 12:46:46.849170 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.849086    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"]
Sep 16 12:46:46.852815 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.852776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:46.855448 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.855430    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5c14b3f0-kserve-self-signed-certs\""
Sep 16 12:46:46.866381 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.866356    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"]
Sep 16 12:46:46.937724 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.937698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-home\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:46.937861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.937754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wmv\" (UniqueName: \"kubernetes.io/projected/7288dbb4-61f4-4b0d-8026-821565ce5666-kube-api-access-z7wmv\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:46.937861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.937838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-tmp-dir\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:46.937953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.937864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7288dbb4-61f4-4b0d-8026-821565ce5666-tls-certs\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:46.937953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.937897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-model-cache\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:46.937953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:46.937928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-dshm\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.038734 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.038710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wmv\" (UniqueName: \"kubernetes.io/projected/7288dbb4-61f4-4b0d-8026-821565ce5666-kube-api-access-z7wmv\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.038865 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.038743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-tmp-dir\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.038865 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.038762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7288dbb4-61f4-4b0d-8026-821565ce5666-tls-certs\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.038865 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.038810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-model-cache\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.038865 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.038831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-dshm\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.038865 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.038862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-home\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.039149 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.039130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-tmp-dir\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.039230 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.039209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-model-cache\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.039294 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.039227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-home\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.041157 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.041138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-dshm\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.041439 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.041423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7288dbb4-61f4-4b0d-8026-821565ce5666-tls-certs\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.048118 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.048092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wmv\" (UniqueName: \"kubernetes.io/projected/7288dbb4-61f4-4b0d-8026-821565ce5666-kube-api-access-z7wmv\") pod \"e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") " pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.162106 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.162024    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:47.435655 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.435626    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"]
Sep 16 12:46:47.437404 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:46:47.437372    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7288dbb4_61f4_4b0d_8026_821565ce5666.slice/crio-70f373fbd0f6803b6bf7dd4eb6043fe222e24a6f27bfbfc83c997a5b79a99c33 WatchSource:0}: Error finding container 70f373fbd0f6803b6bf7dd4eb6043fe222e24a6f27bfbfc83c997a5b79a99c33: Status 404 returned error can't find the container with id 70f373fbd0f6803b6bf7dd4eb6043fe222e24a6f27bfbfc83c997a5b79a99c33
Sep 16 12:46:47.556741 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.556713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb" event={"ID":"7288dbb4-61f4-4b0d-8026-821565ce5666","Type":"ContainerStarted","Data":"ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f"}
Sep 16 12:46:47.556909 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.556750    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb" event={"ID":"7288dbb4-61f4-4b0d-8026-821565ce5666","Type":"ContainerStarted","Data":"70f373fbd0f6803b6bf7dd4eb6043fe222e24a6f27bfbfc83c997a5b79a99c33"}
Sep 16 12:46:47.574738 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:47.574683    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb" podStartSLOduration=1.574663802 podStartE2EDuration="1.574663802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:47.572314296 +0000 UTC m=+2603.707636266" watchObservedRunningTime="2026-09-16 12:46:47.574663802 +0000 UTC m=+2603.709985777"
Sep 16 12:46:48.566951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:48.566919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:48.567465 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:48.566962    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:48.579424 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:48.579405    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:49.576853 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:49.576820    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:46:57.163019 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:57.162988    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:57.163391 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:57.163038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:57.175331 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:57.175305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:46:57.616329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:46:57.616304    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:47:25.021452 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:25.021424    2560 scope.go:117] "RemoveContainer" containerID="d22e63ac31efd7d022e854161da8493a791be1b5ab2ad29b6e7d107b7afff971"
Sep 16 12:47:25.029518 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:25.029499    2560 scope.go:117] "RemoveContainer" containerID="ee88c1878e9e99954ff26e0739466b880b98a64593574746a1f81778f02febdf"
Sep 16 12:47:27.277962 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:27.277931    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"]
Sep 16 12:47:29.145857 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:29.145820    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"]
Sep 16 12:47:42.297058 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.296999    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx" podUID="bedb1bfa-13ef-4141-93e5-fed6cd91cb92" containerName="main" containerID="cri-o://d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce" gracePeriod=45
Sep 16 12:47:42.540695 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.540667    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:47:42.690252 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-model-cache\") pod \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") "
Sep 16 12:47:42.690416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690266    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnwtq\" (UniqueName: \"kubernetes.io/projected/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-kube-api-access-pnwtq\") pod \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") "
Sep 16 12:47:42.690416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690302    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-dshm\") pod \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") "
Sep 16 12:47:42.690416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690331    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-home\") pod \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") "
Sep 16 12:47:42.690416 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690392    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tls-certs\") pod \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") "
Sep 16 12:47:42.690634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690485    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tmp-dir\") pod \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\" (UID: \"bedb1bfa-13ef-4141-93e5-fed6cd91cb92\") "
Sep 16 12:47:42.690634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690547    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-model-cache" (OuterVolumeSpecName: "model-cache") pod "bedb1bfa-13ef-4141-93e5-fed6cd91cb92" (UID: "bedb1bfa-13ef-4141-93e5-fed6cd91cb92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:42.690634 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690558    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-home" (OuterVolumeSpecName: "home") pod "bedb1bfa-13ef-4141-93e5-fed6cd91cb92" (UID: "bedb1bfa-13ef-4141-93e5-fed6cd91cb92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:42.690776 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690716    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bedb1bfa-13ef-4141-93e5-fed6cd91cb92" (UID: "bedb1bfa-13ef-4141-93e5-fed6cd91cb92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:42.690861 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690841    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:42.690903 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690866    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:42.690903 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.690879    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:42.692877 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.692842    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bedb1bfa-13ef-4141-93e5-fed6cd91cb92" (UID: "bedb1bfa-13ef-4141-93e5-fed6cd91cb92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:47:42.693054 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.693036    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-dshm" (OuterVolumeSpecName: "dshm") pod "bedb1bfa-13ef-4141-93e5-fed6cd91cb92" (UID: "bedb1bfa-13ef-4141-93e5-fed6cd91cb92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:42.693103 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.693054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-kube-api-access-pnwtq" (OuterVolumeSpecName: "kube-api-access-pnwtq") pod "bedb1bfa-13ef-4141-93e5-fed6cd91cb92" (UID: "bedb1bfa-13ef-4141-93e5-fed6cd91cb92"). InnerVolumeSpecName "kube-api-access-pnwtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:47:42.791848 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.791827    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnwtq\" (UniqueName: \"kubernetes.io/projected/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-kube-api-access-pnwtq\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:42.791848 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.791846    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:42.791981 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.791855    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedb1bfa-13ef-4141-93e5-fed6cd91cb92-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:42.823276 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.823251    2560 generic.go:356] "Generic (PLEG): container finished" podID="bedb1bfa-13ef-4141-93e5-fed6cd91cb92" containerID="d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce" exitCode=0
Sep 16 12:47:42.823380 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.823316    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"
Sep 16 12:47:42.823380 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.823349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx" event={"ID":"bedb1bfa-13ef-4141-93e5-fed6cd91cb92","Type":"ContainerDied","Data":"d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce"}
Sep 16 12:47:42.823477 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.823398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx" event={"ID":"bedb1bfa-13ef-4141-93e5-fed6cd91cb92","Type":"ContainerDied","Data":"3ceebdaf3c551f1a8ce669556bc00c2461db43e3e2fc8b1ea6ce18b4f4c10578"}
Sep 16 12:47:42.823477 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.823430    2560 scope.go:117] "RemoveContainer" containerID="d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce"
Sep 16 12:47:42.832951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.832935    2560 scope.go:117] "RemoveContainer" containerID="d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce"
Sep 16 12:47:42.833248 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:47:42.833229    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce\": container with ID starting with d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce not found: ID does not exist" containerID="d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce"
Sep 16 12:47:42.833297 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.833256    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce"} err="failed to get container status \"d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce\": rpc error: code = NotFound desc = could not find container \"d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce\": container with ID starting with d07a3945305f0ddb62c1ba7f81c2657d27eece0a69cb6a553955c764242a04ce not found: ID does not exist"
Sep 16 12:47:42.845887 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.845867    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"]
Sep 16 12:47:42.850275 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:42.850255    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5c14b3f0-kserve-cb774c65f-296zx"]
Sep 16 12:47:44.166584 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.166541    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb" podUID="7288dbb4-61f4-4b0d-8026-821565ce5666" containerName="main" containerID="cri-o://ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f" gracePeriod=45
Sep 16 12:47:44.324259 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.324224    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bedb1bfa-13ef-4141-93e5-fed6cd91cb92" path="/var/lib/kubelet/pods/bedb1bfa-13ef-4141-93e5-fed6cd91cb92/volumes"
Sep 16 12:47:44.404262 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.404234    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:47:44.507086 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507028    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7288dbb4-61f4-4b0d-8026-821565ce5666-tls-certs\") pod \"7288dbb4-61f4-4b0d-8026-821565ce5666\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") "
Sep 16 12:47:44.507191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507103    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-home\") pod \"7288dbb4-61f4-4b0d-8026-821565ce5666\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") "
Sep 16 12:47:44.507191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507147    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-model-cache\") pod \"7288dbb4-61f4-4b0d-8026-821565ce5666\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") "
Sep 16 12:47:44.507191 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507177    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-dshm\") pod \"7288dbb4-61f4-4b0d-8026-821565ce5666\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") "
Sep 16 12:47:44.507329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507226    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-tmp-dir\") pod \"7288dbb4-61f4-4b0d-8026-821565ce5666\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") "
Sep 16 12:47:44.507329 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507268    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7wmv\" (UniqueName: \"kubernetes.io/projected/7288dbb4-61f4-4b0d-8026-821565ce5666-kube-api-access-z7wmv\") pod \"7288dbb4-61f4-4b0d-8026-821565ce5666\" (UID: \"7288dbb4-61f4-4b0d-8026-821565ce5666\") "
Sep 16 12:47:44.507428 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507409    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-home" (OuterVolumeSpecName: "home") pod "7288dbb4-61f4-4b0d-8026-821565ce5666" (UID: "7288dbb4-61f4-4b0d-8026-821565ce5666"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:44.507492 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-model-cache" (OuterVolumeSpecName: "model-cache") pod "7288dbb4-61f4-4b0d-8026-821565ce5666" (UID: "7288dbb4-61f4-4b0d-8026-821565ce5666"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:44.507551 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507514    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7288dbb4-61f4-4b0d-8026-821565ce5666" (UID: "7288dbb4-61f4-4b0d-8026-821565ce5666"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:44.507626 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507608    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:44.509129 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.507632    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:44.509246 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.509150    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:44.515727 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.510182    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7288dbb4-61f4-4b0d-8026-821565ce5666-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7288dbb4-61f4-4b0d-8026-821565ce5666" (UID: "7288dbb4-61f4-4b0d-8026-821565ce5666"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:47:44.516347 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.516325    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-dshm" (OuterVolumeSpecName: "dshm") pod "7288dbb4-61f4-4b0d-8026-821565ce5666" (UID: "7288dbb4-61f4-4b0d-8026-821565ce5666"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:47:44.516347 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.516335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7288dbb4-61f4-4b0d-8026-821565ce5666-kube-api-access-z7wmv" (OuterVolumeSpecName: "kube-api-access-z7wmv") pod "7288dbb4-61f4-4b0d-8026-821565ce5666" (UID: "7288dbb4-61f4-4b0d-8026-821565ce5666"). InnerVolumeSpecName "kube-api-access-z7wmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:47:44.609951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.609928    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7288dbb4-61f4-4b0d-8026-821565ce5666-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:44.609951 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.609948    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7wmv\" (UniqueName: \"kubernetes.io/projected/7288dbb4-61f4-4b0d-8026-821565ce5666-kube-api-access-z7wmv\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:44.610078 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.609958    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7288dbb4-61f4-4b0d-8026-821565ce5666-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 16 12:47:44.835763 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.835688    2560 generic.go:356] "Generic (PLEG): container finished" podID="7288dbb4-61f4-4b0d-8026-821565ce5666" containerID="ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f" exitCode=0
Sep 16 12:47:44.835763 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.835750    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"
Sep 16 12:47:44.835936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.835773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb" event={"ID":"7288dbb4-61f4-4b0d-8026-821565ce5666","Type":"ContainerDied","Data":"ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f"}
Sep 16 12:47:44.835936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.835850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb" event={"ID":"7288dbb4-61f4-4b0d-8026-821565ce5666","Type":"ContainerDied","Data":"70f373fbd0f6803b6bf7dd4eb6043fe222e24a6f27bfbfc83c997a5b79a99c33"}
Sep 16 12:47:44.835936 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.835873    2560 scope.go:117] "RemoveContainer" containerID="ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f"
Sep 16 12:47:44.844533 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.844518    2560 scope.go:117] "RemoveContainer" containerID="ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f"
Sep 16 12:47:44.844819 ip-10-0-135-242 kubenswrapper[2560]: E0916 12:47:44.844769    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f\": container with ID starting with ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f not found: ID does not exist" containerID="ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f"
Sep 16 12:47:44.844925 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.844828    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f"} err="failed to get container status \"ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f\": rpc error: code = NotFound desc = could not find container \"ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f\": container with ID starting with ceafe138b13ca9ac5e9a91097d215ae120f373c02395d38e321ec484908f7a0f not found: ID does not exist"
Sep 16 12:47:44.857143 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.857120    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"]
Sep 16 12:47:44.861130 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:44.861111    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5c14b3f0-kserve-5585b7cf99-pfwhb"]
Sep 16 12:47:46.325076 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:47:46.325047    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7288dbb4-61f4-4b0d-8026-821565ce5666" path="/var/lib/kubelet/pods/7288dbb4-61f4-4b0d-8026-821565ce5666/volumes"
Sep 16 12:48:19.074804 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:19.074757    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cc94d66d7-qb2tn_1208d70d-10bc-4fe9-acca-e2034f527fac/authorino/0.log"
Sep 16 12:48:24.529134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:24.529103    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:48:24.540245 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:24.540222    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:48:25.054714 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:25.054685    2560 scope.go:117] "RemoveContainer" containerID="76d1c188a8cccd9dff13d6ba97e4d77653eb941844f20a2e85d485a213d116d3"
Sep 16 12:48:25.301568 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:25.301543    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp_b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1/copy-manifests/0.log"
Sep 16 12:48:25.319620 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:25.319555    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp_b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1/manager/0.log"
Sep 16 12:48:25.923309 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:25.923276    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826030-fqj7p_9ebad9f8-8623-4140-ada5-004a465ef72a/cleanup/0.log"
Sep 16 12:48:26.036328 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:26.036303    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826045-ns75c_607ed29e-1f18-486f-bfd5-8c15b44c74c8/cleanup/0.log"
Sep 16 12:48:26.913738 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:26.913703    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cc94d66d7-qb2tn_1208d70d-10bc-4fe9-acca-e2034f527fac/authorino/0.log"
Sep 16 12:48:28.212503 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:28.212471    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-96rxq_df1ff7b3-88ce-46cf-bdbe-f45c472f0458/kube-auth-proxy/0.log"
Sep 16 12:48:28.681851 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:28.681815    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8hfz8_8120361a-3d74-47be-b9a8-5ece86158766/payload-processing/0.log"
Sep 16 12:48:28.799311 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:28.799281    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb97b79b-6jf4w_bec56780-8565-43f0-9159-57e83ca48bed/router/0.log"
Sep 16 12:48:29.252135 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:29.252106    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-p4dsx_a023d8a6-4d8b-4c46-aa4b-40b876a5ee62/main/0.log"
Sep 16 12:48:29.471147 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:29.471116    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8nt8x_8094939c-9778-49ec-b2d7-e72f397086b9/main/0.log"
Sep 16 12:48:29.577737 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:29.577648    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqb2r9_20529458-de4e-41e5-bd25-82e02555b099/main/0.log"
Sep 16 12:48:29.798426 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:29.798395    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zgs8s_7ceb40ee-1e7a-427f-8276-ff035f7f89c2/main/0.log"
Sep 16 12:48:35.269042 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.269010    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2psvv/must-gather-jbrww"]
Sep 16 12:48:35.269415 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.269399    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bedb1bfa-13ef-4141-93e5-fed6cd91cb92" containerName="main"
Sep 16 12:48:35.269415 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.269416    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bedb1bfa-13ef-4141-93e5-fed6cd91cb92" containerName="main"
Sep 16 12:48:35.269557 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.269425    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7288dbb4-61f4-4b0d-8026-821565ce5666" containerName="main"
Sep 16 12:48:35.269557 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.269433    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7288dbb4-61f4-4b0d-8026-821565ce5666" containerName="main"
Sep 16 12:48:35.269664 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.269573    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bedb1bfa-13ef-4141-93e5-fed6cd91cb92" containerName="main"
Sep 16 12:48:35.269664 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.269593    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7288dbb4-61f4-4b0d-8026-821565ce5666" containerName="main"
Sep 16 12:48:35.271587 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.271560    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.274220 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.274200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2psvv\"/\"default-dockercfg-nq47s\""
Sep 16 12:48:35.274775 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.274761    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2psvv\"/\"openshift-service-ca.crt\""
Sep 16 12:48:35.275094 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.275080    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2psvv\"/\"kube-root-ca.crt\""
Sep 16 12:48:35.286676 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.286655    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/must-gather-jbrww"]
Sep 16 12:48:35.431781 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.431752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a7d1602-37a5-431c-93a6-faaa3b958c8e-must-gather-output\") pod \"must-gather-jbrww\" (UID: \"4a7d1602-37a5-431c-93a6-faaa3b958c8e\") " pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.431953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.431835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv57\" (UniqueName: \"kubernetes.io/projected/4a7d1602-37a5-431c-93a6-faaa3b958c8e-kube-api-access-ktv57\") pod \"must-gather-jbrww\" (UID: \"4a7d1602-37a5-431c-93a6-faaa3b958c8e\") " pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.532870 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.532810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a7d1602-37a5-431c-93a6-faaa3b958c8e-must-gather-output\") pod \"must-gather-jbrww\" (UID: \"4a7d1602-37a5-431c-93a6-faaa3b958c8e\") " pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.532870 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.532850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv57\" (UniqueName: \"kubernetes.io/projected/4a7d1602-37a5-431c-93a6-faaa3b958c8e-kube-api-access-ktv57\") pod \"must-gather-jbrww\" (UID: \"4a7d1602-37a5-431c-93a6-faaa3b958c8e\") " pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.533113 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.533095    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a7d1602-37a5-431c-93a6-faaa3b958c8e-must-gather-output\") pod \"must-gather-jbrww\" (UID: \"4a7d1602-37a5-431c-93a6-faaa3b958c8e\") " pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.541774 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.541749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv57\" (UniqueName: \"kubernetes.io/projected/4a7d1602-37a5-431c-93a6-faaa3b958c8e-kube-api-access-ktv57\") pod \"must-gather-jbrww\" (UID: \"4a7d1602-37a5-431c-93a6-faaa3b958c8e\") " pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.580868 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.580843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/must-gather-jbrww"
Sep 16 12:48:35.856052 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.856023    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/must-gather-jbrww"]
Sep 16 12:48:35.858498 ip-10-0-135-242 kubenswrapper[2560]: W0916 12:48:35.858473    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7d1602_37a5_431c_93a6_faaa3b958c8e.slice/crio-7c85c47c6300153266c9cc9397d1de9f00b11cb71cba9912e18f6d8e05944a22 WatchSource:0}: Error finding container 7c85c47c6300153266c9cc9397d1de9f00b11cb71cba9912e18f6d8e05944a22: Status 404 returned error can't find the container with id 7c85c47c6300153266c9cc9397d1de9f00b11cb71cba9912e18f6d8e05944a22
Sep 16 12:48:35.860557 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:35.860534    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:48:36.084144 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:36.084111    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/must-gather-jbrww" event={"ID":"4a7d1602-37a5-431c-93a6-faaa3b958c8e","Type":"ContainerStarted","Data":"7c85c47c6300153266c9cc9397d1de9f00b11cb71cba9912e18f6d8e05944a22"}
Sep 16 12:48:37.093162 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:37.093121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/must-gather-jbrww" event={"ID":"4a7d1602-37a5-431c-93a6-faaa3b958c8e","Type":"ContainerStarted","Data":"d5c721de20907657564728b595e2f9f11ed18aaa59efc27a6da833b45681cf1e"}
Sep 16 12:48:38.103240 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:38.103188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/must-gather-jbrww" event={"ID":"4a7d1602-37a5-431c-93a6-faaa3b958c8e","Type":"ContainerStarted","Data":"29942902925da307788f6ac3ffaca2259e1309f88a20fa1f1a0d1965813110d6"}
Sep 16 12:48:38.566320 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:38.566245    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pvbqc_a4698db6-e801-4511-8142-6a53c3338306/global-pull-secret-syncer/0.log"
Sep 16 12:48:38.769053 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:38.769018    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lzfbc_d95554cf-2a64-4e58-8e24-275ebf9ffb78/konnectivity-agent/0.log"
Sep 16 12:48:38.797277 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:38.797245    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-242.ec2.internal_cc20f8ec2b2204fa94263ebc59d562b0/haproxy/0.log"
Sep 16 12:48:43.219808 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:43.219065    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cc94d66d7-qb2tn_1208d70d-10bc-4fe9-acca-e2034f527fac/authorino/0.log"
Sep 16 12:48:45.227904 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.227818    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-22g9k_8dd8f8b9-4b8b-45f4-bb54-d14ba2cb68bc/cluster-monitoring-operator/0.log"
Sep 16 12:48:45.257357 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.257324    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-ftbc8_9daf85a9-d24f-4b15-b09b-868e5999d3db/kube-state-metrics/0.log"
Sep 16 12:48:45.282033 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.282002    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-ftbc8_9daf85a9-d24f-4b15-b09b-868e5999d3db/kube-rbac-proxy-main/0.log"
Sep 16 12:48:45.305779 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.305744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-ftbc8_9daf85a9-d24f-4b15-b09b-868e5999d3db/kube-rbac-proxy-self/0.log"
Sep 16 12:48:45.495098 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.495019    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkpnv_b4eae29c-5100-4f8e-ae4a-5cfba555e551/node-exporter/0.log"
Sep 16 12:48:45.535917 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.535873    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkpnv_b4eae29c-5100-4f8e-ae4a-5cfba555e551/kube-rbac-proxy/0.log"
Sep 16 12:48:45.617209 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.617178    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkpnv_b4eae29c-5100-4f8e-ae4a-5cfba555e551/init-textfile/0.log"
Sep 16 12:48:45.923493 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.923213    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51206dfa-8638-4f66-ac3c-dfdbd22ff570/prometheus/0.log"
Sep 16 12:48:45.952542 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.952515    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51206dfa-8638-4f66-ac3c-dfdbd22ff570/config-reloader/0.log"
Sep 16 12:48:45.978121 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:45.978079    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51206dfa-8638-4f66-ac3c-dfdbd22ff570/thanos-sidecar/0.log"
Sep 16 12:48:46.027354 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.027313    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51206dfa-8638-4f66-ac3c-dfdbd22ff570/kube-rbac-proxy-web/0.log"
Sep 16 12:48:46.052751 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.052725    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51206dfa-8638-4f66-ac3c-dfdbd22ff570/kube-rbac-proxy/0.log"
Sep 16 12:48:46.080449 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.080424    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51206dfa-8638-4f66-ac3c-dfdbd22ff570/kube-rbac-proxy-thanos/0.log"
Sep 16 12:48:46.108835 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.108812    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51206dfa-8638-4f66-ac3c-dfdbd22ff570/init-config-reloader/0.log"
Sep 16 12:48:46.144236 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.144210    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-nvqbq_50e827b4-8130-4a82-bbd4-3fab17dc154c/prometheus-operator/0.log"
Sep 16 12:48:46.173688 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.173602    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-nvqbq_50e827b4-8130-4a82-bbd4-3fab17dc154c/kube-rbac-proxy/0.log"
Sep 16 12:48:46.338679 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.338647    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b474c465-8rpdm_2d16564e-095d-4135-b3a0-78f92c472b35/thanos-query/0.log"
Sep 16 12:48:46.361728 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.361701    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b474c465-8rpdm_2d16564e-095d-4135-b3a0-78f92c472b35/kube-rbac-proxy-web/0.log"
Sep 16 12:48:46.385315 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.385285    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b474c465-8rpdm_2d16564e-095d-4135-b3a0-78f92c472b35/kube-rbac-proxy/0.log"
Sep 16 12:48:46.406984 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.406956    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b474c465-8rpdm_2d16564e-095d-4135-b3a0-78f92c472b35/prom-label-proxy/0.log"
Sep 16 12:48:46.435313 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.435241    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b474c465-8rpdm_2d16564e-095d-4135-b3a0-78f92c472b35/kube-rbac-proxy-rules/0.log"
Sep 16 12:48:46.457669 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:46.457639    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75b474c465-8rpdm_2d16564e-095d-4135-b3a0-78f92c472b35/kube-rbac-proxy-metrics/0.log"
Sep 16 12:48:47.933653 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:47.933609    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-j52wh_c8aa3fd1-fdb8-4f06-82f4-01dfe5c84f90/networking-console-plugin/0.log"
Sep 16 12:48:48.547403 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.547353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/1.log"
Sep 16 12:48:48.556410 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.556385    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rkmf9_0bc44c33-1937-4555-bbe5-de6ea2a19c8c/console-operator/2.log"
Sep 16 12:48:48.760088 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.760019    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2psvv/must-gather-jbrww" podStartSLOduration=12.682046955 podStartE2EDuration="13.75999487s" totalImagesPullingTime="1.077947915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:38.122987953 +0000 UTC m=+2714.258309927" watchObservedRunningTime="2026-09-16 12:48:48.75999487 +0000 UTC m=+2724.895316841"
Sep 16 12:48:48.761548 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.761521    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"]
Sep 16 12:48:48.769546 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.769523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.776333 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.776285    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"]
Sep 16 12:48:48.878141 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.878085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-podres\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.878343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.878214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-lib-modules\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.878343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.878253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-sys\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.878343 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.878295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-proc\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.878509 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.878354    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqxt\" (UniqueName: \"kubernetes.io/projected/30f77326-ba5a-4a18-8220-b104f3b96342-kube-api-access-zlqxt\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979292 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-lib-modules\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-sys\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-proc\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-sys\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqxt\" (UniqueName: \"kubernetes.io/projected/30f77326-ba5a-4a18-8220-b104f3b96342-kube-api-access-zlqxt\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-lib-modules\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-proc\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-podres\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.979800 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.979685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30f77326-ba5a-4a18-8220-b104f3b96342-podres\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:48.988732 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:48.988700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqxt\" (UniqueName: \"kubernetes.io/projected/30f77326-ba5a-4a18-8220-b104f3b96342-kube-api-access-zlqxt\") pod \"perf-node-gather-daemonset-lq26n\" (UID: \"30f77326-ba5a-4a18-8220-b104f3b96342\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:49.083103 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:49.083067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:49.375315 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:49.375284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"]
Sep 16 12:48:50.210192 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:50.210156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n" event={"ID":"30f77326-ba5a-4a18-8220-b104f3b96342","Type":"ContainerStarted","Data":"012c29cc9c3c26ae40a3403b30d4e257e634d41254845980ea489e2c49eea4a9"}
Sep 16 12:48:50.210580 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:50.210219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n" event={"ID":"30f77326-ba5a-4a18-8220-b104f3b96342","Type":"ContainerStarted","Data":"49eaa325df1aa8832a7554ff342ea68267315f017b4fe77fd584ef6b51f9163d"}
Sep 16 12:48:50.210580 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:50.210246    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:50.229994 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:50.229938    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n" podStartSLOduration=2.229918814 podStartE2EDuration="2.229918814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:50.228930666 +0000 UTC m=+2726.364252638" watchObservedRunningTime="2026-09-16 12:48:50.229918814 +0000 UTC m=+2726.365240787"
Sep 16 12:48:51.018164 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:51.018134    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-74nwl_97a89fdd-e02c-454c-8a5d-787e2b5a5c8f/dns/0.log"
Sep 16 12:48:51.039457 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:51.039372    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-74nwl_97a89fdd-e02c-454c-8a5d-787e2b5a5c8f/kube-rbac-proxy/0.log"
Sep 16 12:48:51.168515 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:51.168478    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7lbrw_ec9781e9-d23e-44bb-a141-1fa6790be59b/dns-node-resolver/0.log"
Sep 16 12:48:51.789613 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:51.789583    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbjcp_205eab7e-100c-46e7-b39b-e3aef2be4adf/node-ca/0.log"
Sep 16 12:48:53.320408 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:53.320375    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-96rxq_df1ff7b3-88ce-46cf-bdbe-f45c472f0458/kube-auth-proxy/0.log"
Sep 16 12:48:53.458952 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:53.458919    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8hfz8_8120361a-3d74-47be-b9a8-5ece86158766/payload-processing/0.log"
Sep 16 12:48:53.482884 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:53.482852    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb97b79b-6jf4w_bec56780-8565-43f0-9159-57e83ca48bed/router/0.log"
Sep 16 12:48:54.381405 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:54.381374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8cgc_7520f5cb-0f5e-417c-8c1a-e3311b7c23ee/serve-healthcheck-canary/0.log"
Sep 16 12:48:54.894953 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:54.894921    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kdfjs_a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24/insights-operator/0.log"
Sep 16 12:48:54.895761 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:54.895733    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kdfjs_a81fa552-7a8e-4ae4-9c7a-ad8f11ae1a24/insights-operator/1.log"
Sep 16 12:48:54.918090 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:54.918065    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7bd29_90740c99-e643-46f8-8d06-b9c67ab4b81b/kube-rbac-proxy/0.log"
Sep 16 12:48:54.951125 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:54.951101    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7bd29_90740c99-e643-46f8-8d06-b9c67ab4b81b/exporter/0.log"
Sep 16 12:48:55.018215 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:55.018177    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7bd29_90740c99-e643-46f8-8d06-b9c67ab4b81b/extractor/0.log"
Sep 16 12:48:56.226764 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:56.226737    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-lq26n"
Sep 16 12:48:58.142614 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:58.142579    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp_b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1/manager/0.log"
Sep 16 12:48:58.159841 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:58.159781    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-tgnbp_b36796c9-c3a4-4d67-a94f-bd8e0f9fdbd1/copy-manifests/0.log"
Sep 16 12:48:59.523122 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:59.523092    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-96rxq_df1ff7b3-88ce-46cf-bdbe-f45c472f0458/kube-auth-proxy/0.log"
Sep 16 12:48:59.547717 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:59.547687    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74d55495b7-hrcxp_9b5848b3-575a-4d5d-8562-1ce29a0182b8/manager/0.log"
Sep 16 12:48:59.617332 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:59.617300    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8hfz8_8120361a-3d74-47be-b9a8-5ece86158766/payload-processing/0.log"
Sep 16 12:48:59.636255 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:48:59.636218    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb97b79b-6jf4w_bec56780-8565-43f0-9159-57e83ca48bed/router/0.log"
Sep 16 12:49:01.569134 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:49:01.569105    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f67fb6dbc-96rxq_df1ff7b3-88ce-46cf-bdbe-f45c472f0458/kube-auth-proxy/0.log"
Sep 16 12:49:01.660898 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:49:01.660866    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8hfz8_8120361a-3d74-47be-b9a8-5ece86158766/payload-processing/0.log"
Sep 16 12:49:01.675178 ip-10-0-135-242 kubenswrapper[2560]: I0916 12:49:01.675152    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb97b79b-6jf4w_bec56780-8565-43f0-9159-57e83ca48bed/router/0.log"